2024-03-24T00:13:59.571 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T00:13:59.572 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} --owner mchangir --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438 -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/orig.config.yaml 2024-03-24T00:13:59.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:13:59.728 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} email: mchangir@redhat.com first_in_suite: false job_id: '7616438' kernel: client: branch: testing flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.dir_split install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJd8CcRmbMKD+QnCRekSe6ibcqyE+2ZUHS11/ROZXhpjiZrz37YK1GE6NDNCcvcBMGDCt5vn2+yAxavfighCAM= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO1wHQDpPl2GQcDlVWxBrF7Adx1vNkAJVFxTQGHT2vC5+wJLcYHzW7O/lyCn4IfCKMy6l6h1ROCtqPjUB5+vgfA= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK5nwh+c/dz03Tzy3YnN8bnfvfGFcbblgZcdk9qDpoPMYBkwVJa5gaq6d/CpJcUeNTzrzVNsMXYfBaeYRQiO1w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T00:13:59.728 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-24T00:13:59.729 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-24T00:13:59.729 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T00:13:59.731 INFO:teuthology.task.internal:Checking packages... 2024-03-24T00:13:59.755 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-24T00:13:59.755 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T00:13:59.755 INFO:teuthology.packaging:ref: None 2024-03-24T00:13:59.755 INFO:teuthology.packaging:tag: None 2024-03-24T00:13:59.755 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-24T00:13:59.755 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:13:59.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-24T00:13:59.978 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-24T00:13:59.980 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T00:13:59.990 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T00:13:59.990 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T00:14:00.001 INFO:teuthology.task.internal:Saving configuration 2024-03-24T00:14:00.023 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T00:14:00.030 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:14:00.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.766478', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJd8CcRmbMKD+QnCRekSe6ibcqyE+2ZUHS11/ROZXhpjiZrz37YK1GE6NDNCcvcBMGDCt5vn2+yAxavfighCAM='} 2024-03-24T00:14:00.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.768068', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO1wHQDpPl2GQcDlVWxBrF7Adx1vNkAJVFxTQGHT2vC5+wJLcYHzW7O/lyCn4IfCKMy6l6h1ROCtqPjUB5+vgfA='} 2024-03-24T00:14:00.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.764899', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK5nwh+c/dz03Tzy3YnN8bnfvfGFcbblgZcdk9qDpoPMYBkwVJa5gaq6d/CpJcUeNTzrzVNsMXYfBaeYRQiO1w='} 2024-03-24T00:14:00.093 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T00:14:00.099 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-24T00:14:00.099 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-24T00:14:00.099 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-24T00:14:00.099 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T00:14:00.186 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c8eb2af70>, signals=[15]) 2024-03-24T00:14:00.187 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T00:14:00.193 INFO:teuthology.task.internal:Opening connections... 2024-03-24T00:14:00.193 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:14:00.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:14:00.290 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-24T00:14:00.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:14:00.375 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-24T00:14:00.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:14:00.463 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T00:14:00.470 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-03-24T00:14:00.489 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-03-24T00:14:00.489 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-03-24T00:14:00.546 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-03-24T00:14:00.546 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-03-24T00:14:00.546 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T00:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T00:14:00.548 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-03-24T00:14:00.570 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-24T00:14:00.588 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-24T00:14:00.588 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-24T00:14:00.647 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-24T00:14:00.648 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-24T00:14:00.648 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-24T00:14:00.648 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-24T00:14:00.648 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-24T00:14:00.648 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T00:14:00.649 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T00:14:00.649 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-24T00:14:00.678 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-24T00:14:00.697 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-24T00:14:00.697 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-24T00:14:00.752 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T00:14:00.753 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T00:14:00.754 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T00:14:00.754 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-24T00:14:00.778 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T00:14:00.787 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T00:14:00.815 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T00:14:00.815 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:14:00.817 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:14:00.820 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:14:00.836 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T00:14:00.840 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T00:14:00.840 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:14:00.874 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:14:00.876 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:14:00.967 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T00:14:01.035 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T00:14:01.049 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-24T00:14:01.050 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-24T00:14:01.050 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-24T00:14:01.050 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:14:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:14:01.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-24T00:14:01.359 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:14:01.360 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T00:14:01.360 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-24T00:14:01.377 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-361.el9.x86_64 2024-03-24T00:14:01.377 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:14:01.378 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T00:14:01.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum upgrade -y linux-firmware 2024-03-24T00:14:03.485 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-24T00:14:04.877 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T00:14:05.945 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T00:14:07.635 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-24T00:14:11.763 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-24T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-24T00:14:14.915 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:14:19.871 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T00:14:22.063 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-24T00:14:22.071 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 1 Package 2024-03-24T00:14:22.072 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:14:22.072 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 375 M 2024-03-24T00:14:22.072 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-24T00:14:28.738 INFO:teuthology.orchestra.run.smithi123.stdout:linux-firmware-20240219-143.el9.noarch.rpm 56 MB/s | 375 MB 00:06 2024-03-24T00:14:28.739 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:14:28.740 INFO:teuthology.orchestra.run.smithi123.stdout:Total 56 MB/s | 375 MB 00:06 2024-03-24T00:14:28.740 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-24T00:14:28.835 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-24T00:14:28.835 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-24T00:14:31.198 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-24T00:14:31.198 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-24T00:14:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-24T00:14:37.824 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T00:14:37.904 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T00:14:39.269 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T00:14:39.269 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T00:14:39.616 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T00:14:39.617 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:14:39.617 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-03-24T00:14:39.617 INFO:teuthology.orchestra.run.smithi123.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-24T00:14:39.617 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:14:39.617 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-24T00:14:39.913 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-24T00:14:39.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:14:40.066 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-24T00:14:40.066 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-24T00:14:40.069 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi123.front.sepia.ceph.com to complete... 2024-03-24T00:14:51.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24 00:14:51 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-24T00:14:51.481 INFO:teuthology.orchestra.run.smithi123.stderr:FINISHED --2024-03-24 00:14:51-- 2024-03-24T00:14:51.481 INFO:teuthology.orchestra.run.smithi123.stderr:Total wall clock time: 11s 2024-03-24T00:14:51.481 INFO:teuthology.orchestra.run.smithi123.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-24T00:14:51.484 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-24T00:14:51.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-24T00:14:53.421 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying... ######################################## 2024-03-24T00:14:53.550 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing... ######################################## 2024-03-24T00:14:53.631 INFO:teuthology.orchestra.run.smithi123.stdout:Updating / installing... 2024-03-24T00:15:01.176 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-24T00:15:16.728 INFO:teuthology.orchestra.run.smithi123.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T00:15:39.069 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-24T00:15:39.069 DEBUG:teuthology.orchestra.run.smithi123:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-24T00:15:39.141 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T00:15:39.142 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-24T00:15:39.143 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T00:15:39.144 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T00:15:39.145 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T00:15:39.146 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-24T00:15:39.147 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T00:15:39.148 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T00:15:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T00:15:39.150 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T00:15:39.151 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T00:15:39.152 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-24T00:15:39.153 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T00:15:39.154 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T00:15:39.155 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-24T00:15:39.156 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T00:15:39.157 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T00:15:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T00:15:39.159 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T00:15:39.160 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T00:15:39.161 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T00:15:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T00:15:39.163 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-24T00:15:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T00:15:39.165 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T00:15:39.166 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T00:15:39.167 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T00:15:39.168 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T00:15:39.169 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T00:15:39.170 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T00:15:39.171 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T00:15:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T00:15:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T00:15:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T00:15:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T00:15:39.176 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-24T00:15:39.177 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T00:15:39.178 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-24T00:15:39.179 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T00:15:39.180 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T00:15:39.181 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T00:15:39.182 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T00:15:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-24T00:15:39.184 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T00:15:39.185 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-24T00:15:39.186 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-24T00:15:39.187 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T00:15:39.188 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T00:15:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T00:15:39.190 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-24T00:15:39.191 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-24T00:15:39.192 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-24T00:15:39.193 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-24T00:15:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T00:15:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T00:15:39.196 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T00:15:39.197 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T00:15:39.198 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-24T00:15:39.199 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-24T00:15:39.200 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-24T00:15:39.201 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T00:15:39.202 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T00:15:39.203 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-24T00:15:39.204 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-24T00:15:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T00:15:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T00:15:39.207 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T00:15:39.208 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T00:15:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T00:15:39.210 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T00:15:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-24T00:15:39.212 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-24T00:15:39.213 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T00:15:39.214 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T00:15:39.215 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-24T00:15:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T00:15:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T00:15:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T00:15:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-24T00:15:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T00:15:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T00:15:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T00:15:39.224 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T00:15:39.225 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T00:15:39.226 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T00:15:39.227 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T00:15:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T00:15:39.229 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-24T00:15:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T00:15:39.231 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T00:15:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-24T00:15:39.233 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T00:15:39.234 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-24T00:15:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T00:15:39.236 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T00:15:39.237 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T00:15:39.238 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T00:15:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T00:15:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T00:15:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T00:15:39.242 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-24T00:15:39.244 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-24T00:15:39.245 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-24T00:15:39.247 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-24T00:15:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T00:15:39.249 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T00:15:39.250 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T00:15:39.251 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T00:15:39.252 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T00:15:39.253 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T00:15:39.254 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T00:15:39.255 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-24T00:15:39.256 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T00:15:39.257 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T00:15:39.258 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T00:15:39.259 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-24T00:15:39.260 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T00:15:39.261 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T00:15:39.262 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T00:15:39.263 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T00:15:39.264 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T00:15:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T00:15:39.266 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T00:15:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T00:15:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T00:15:39.269 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T00:15:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T00:15:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T00:15:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T00:15:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T00:15:39.274 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T00:15:39.275 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-24T00:15:39.276 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T00:15:39.277 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T00:15:39.278 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T00:15:39.278 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T00:15:39.278 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T00:15:39.278 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T00:15:39.278 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T00:15:39.282 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:39.282 DEBUG:teuthology.orchestra.run.smithi123:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-24T00:15:39.303 INFO:teuthology.orchestra.run.smithi123.stdout:postinstall scriptlet (using /bin/sh): 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:for file in vmlinuz System.map config; do 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout: fi 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:39.304 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-24T00:15:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-24T00:15:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-24T00:15:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-24T00:15:39.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-03-24T00:15:39.380 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-03-24T00:15:39.380 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-03-24T00:15:39.380 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-03-24T00:15:39.380 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 68.el9 2024-03-24T00:15:39.380 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8264254 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-03-24T00:15:39.381 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:15:39.382 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T00:15:39.383 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T00:15:39.383 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:39.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T00:15:39.985 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-03-24T00:15:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T00:15:41.449 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-03-24T00:15:41.467 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-24T00:15:41.481 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.Mh8URtxCAd 2024-03-24T00:15:41.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Mh8URtxCAd 2024-03-24T00:15:41.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.Mh8URtxCAd 2024-03-24T00:15:41.690 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.Mh8URtxCAd is 6KB 2024-03-24T00:15:41.709 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.Mh8URtxCAd 2024-03-24T00:15:41.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T00:15:41.787 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T00:15:41.787 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T00:15:41.787 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T00:15:41.787 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-24T00:15:41.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-24T00:15:41.970 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-03-24T00:16:11.975 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-24T00:16:11.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:16:11.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:16:30.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-03-24T00:16:39.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:16:39.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:16:42.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-03-24T00:16:54.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:16:54.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:17:26.378 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-24T00:17:27.057 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:17:27.057 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T00:17:27.057 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-24T00:17:27.095 INFO:teuthology.orchestra.run.smithi123.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:27.096 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:27.096 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:17:27.096 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T00:17:28.097 DEBUG:teuthology.parallel:result is None 2024-03-24T00:17:28.097 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T00:17:28.108 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T00:17:28.108 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:17:28.116 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:17:28.119 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:17:28.165 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T00:17:28.171 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T00:17:28.178 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T00:17:28.178 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:17:28.183 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:17:28.186 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:17:28.332 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T00:17:28.338 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T00:17:28.338 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:17:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:17:28.355 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:17:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:17:28.372 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:17:28.376 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:17:28.397 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:17:28.433 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.450 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.469 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.483 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.487 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.569 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:17:28.572 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T00:17:28.580 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T00:17:28.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:17:28.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:17:28.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:17:28.651 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T00:17:28.662 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T00:17:28.662 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:17:28.667 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:17:28.669 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:17:28.698 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:28.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:28.906 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:28.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:29.061 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:17:29.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:17:29.147 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:29.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:29.245 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:29.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:29.374 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:17:29.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:17:29.442 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:29.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:17:29.546 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:29.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:17:29.631 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:17:29.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:17:29.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-03-24T00:17:29.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-24T00:17:29.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-24T00:17:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:17:29.781 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:17:29.810 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:17:30.242 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T00:17:30.250 INFO:teuthology.task.internal:Starting timer... 2024-03-24T00:17:30.250 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T00:17:30.260 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T00:17:30.268 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T00:17:30.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-03-24T00:17:30.400 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-03-24T00:17:30.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-24T00:17:30.461 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-24T00:17:30.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-24T00:17:30.534 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-24T00:17:30.536 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T00:17:30.536 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-03-24T00:17:30.598 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-03-24T00:17:30.598 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-24T00:17:30.623 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-24T00:17:30.623 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-24T00:17:30.647 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-24T00:17:30.647 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-24T00:17:30.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:17:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:17:30.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:17:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:17:30.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:17:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:17:30.785 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T00:17:30.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:17:30.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:17:30.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:17:30.949 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T00:17:30.958 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T00:17:30.965 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T00:17:31.100 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T00:17:31.139 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T00:17:31.140 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi177.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T00:23:20.482 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-24T00:23:20.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:23:20.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:23:20.633 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-24T00:23:20.792 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-24T00:23:20.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-24T00:23:20.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:23:20.881 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-24T00:23:20.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-24T00:23:20.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-24T00:23:20.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:23:21.041 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-24T00:23:21.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-24T00:23:21.117 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T00:23:21.140 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T00:23:21.141 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:23:21.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:23:21.149 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:23:21.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:23:21.152 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:23:21.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:23:21.186 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:23:21.192 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:23:21.204 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:23:21.209 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:23:21.234 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-24T00:23:21.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-24T00:23:21.247 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:23:21.248 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-24T00:23:21.254 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-24T00:23:21.265 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:23:21.271 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:23:21.282 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:23:21.287 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:23:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:23:21.326 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.331 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.338 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.389 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-03-24T00:23:21.431 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-03-24T00:23:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:23:21.531 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:23:21.656 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:23:21.665 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:23:21.665 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-24T00:23:21.665 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.666 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.666 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.666 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:23:21.668 INFO:teuthology.run_tasks:Running task install... 2024-03-24T00:23:21.678 DEBUG:teuthology.task.install:project ceph 2024-03-24T00:23:21.679 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T00:23:21.679 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T00:23:21.679 INFO:teuthology.task.install:Using flavor: default 2024-03-24T00:23:21.686 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T00:23:21.687 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T00:23:21.687 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:23:21.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:23:21.690 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:23:21.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:23:21.692 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:23:21.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:23:21.839 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:23:21.839 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:23:21.911 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:23:21.911 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:23:21.922 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:23:21.922 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:23:21.950 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:23:21.950 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:23:21.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:23:22.024 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:23:22.024 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:23:22.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:23:22.051 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:23:22.051 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:23:22.053 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:23:22.053 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:23:22.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:23:22.090 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:23:22.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:23:22.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:23:22.124 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:23:22.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:23:22.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:23:22.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:23:22.207 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:23:22.275 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-03-24T00:23:22.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-03-24T00:23:22.279 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:23:22.321 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:23:22.349 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-03-24T00:23:22.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-03-24T00:23:22.407 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-03-24T00:23:22.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-03-24T00:23:22.513 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-03-24T00:23:22.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:23:22.590 INFO:teuthology.orchestra.run.smithi204.stdout:74 files removed 2024-03-24T00:23:22.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:23:22.764 INFO:teuthology.orchestra.run.smithi123.stdout:74 files removed 2024-03-24T00:23:22.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:23:23.200 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 225 kB/s | 83 kB 00:00 2024-03-24T00:23:23.265 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 233 kB/s | 83 kB 00:00 2024-03-24T00:23:23.625 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-24T00:23:23.625 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T00:23:23.882 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 147 kB/s | 83 kB 00:00 2024-03-24T00:23:24.034 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-24T00:23:24.051 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-24T00:23:24.415 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-24T00:23:24.453 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T00:23:24.494 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T00:23:24.916 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-24T00:23:25.493 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T00:23:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-24T00:23:25.944 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-24T00:23:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-24T00:23:26.886 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T00:23:27.112 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-24T00:23:28.540 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.0 MB 00:01 2024-03-24T00:23:28.820 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-24T00:23:28.978 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-24T00:23:39.512 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 866 kB/s | 6.1 MB 00:07 2024-03-24T00:23:39.936 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 835 kB/s | 6.1 MB 00:07 2024-03-24T00:23:41.346 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-24T00:23:41.495 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-24T00:23:42.347 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-24T00:23:42.975 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:23:47.289 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T00:23:47.873 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-24T00:23:49.618 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:23:49.619 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:23:49.633 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:23:49.634 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:23:49.635 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:23:49.636 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:23:49.638 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:23:49.641 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:23:49.642 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:23:49.722 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-24T00:23:49.739 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:23:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:23:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:23:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:23:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:23:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:23:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:23:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:23:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:23:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:23:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:23:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:23:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:23:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:23:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:23:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:23:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:23:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:23:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout:Install 150 Packages 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 24 Packages 2024-03-24T00:23:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:23:49.759 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 224 M 2024-03-24T00:23:49.759 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-24T00:23:50.195 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:23:50.196 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:23:50.210 INFO:teuthology.orchestra.run.smithi204.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:23:50.212 INFO:teuthology.orchestra.run.smithi204.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:23:50.213 INFO:teuthology.orchestra.run.smithi204.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:23:50.214 INFO:teuthology.orchestra.run.smithi204.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:23:50.216 INFO:teuthology.orchestra.run.smithi204.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:23:50.219 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:23:50.220 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:23:50.302 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:23:50.330 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:23:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:23:50.332 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:23:50.333 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:23:50.334 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:23:50.335 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:23:50.336 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:23:50.337 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:23:50.338 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:23:50.339 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:23:50.340 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:23:50.341 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:23:50.342 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:Install 150 Packages 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 24 Packages 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:23:50.343 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 224 M 2024-03-24T00:23:50.344 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-24T00:23:50.526 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 927 kB/s | 19 MB 00:20 2024-03-24T00:23:50.606 INFO:teuthology.orchestra.run.smithi177.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-24T00:23:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-24T00:23:50.940 INFO:teuthology.orchestra.run.smithi204.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-24T00:23:50.957 INFO:teuthology.orchestra.run.smithi177.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.8 MB/s | 145 kB 00:00 2024-03-24T00:23:51.215 INFO:teuthology.orchestra.run.smithi177.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.9 MB/s | 2.3 MB 00:00 2024-03-24T00:23:51.249 INFO:teuthology.orchestra.run.smithi177.stdout:(5/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-24T00:23:51.449 INFO:teuthology.orchestra.run.smithi177.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-03-24T00:23:51.491 INFO:teuthology.orchestra.run.smithi204.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 6.4 MB/s | 5.4 MB 00:00 2024-03-24T00:23:51.566 INFO:teuthology.orchestra.run.smithi204.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.9 MB/s | 145 kB 00:00 2024-03-24T00:23:51.925 INFO:teuthology.orchestra.run.smithi204.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 6.4 MB/s | 2.3 MB 00:00 2024-03-24T00:23:52.166 INFO:teuthology.orchestra.run.smithi204.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-24T00:23:52.808 INFO:teuthology.orchestra.run.smithi204.stdout:(6/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 477 kB/s | 890 kB 00:01 2024-03-24T00:23:52.875 INFO:teuthology.orchestra.run.smithi204.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 6.6 MB/s | 4.7 MB 00:00 2024-03-24T00:23:52.924 INFO:teuthology.orchestra.run.smithi177.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 17 MB 00:01 2024-03-24T00:23:53.916 INFO:teuthology.orchestra.run.smithi177.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 9.9 MB/s | 9.9 MB 00:00 2024-03-24T00:23:53.974 INFO:teuthology.orchestra.run.smithi177.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 432 kB/s | 25 kB 00:00 2024-03-24T00:23:54.042 INFO:teuthology.orchestra.run.smithi204.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 8.5 MB/s | 9.9 MB 00:01 2024-03-24T00:23:54.126 INFO:teuthology.orchestra.run.smithi204.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 302 kB/s | 25 kB 00:00 2024-03-24T00:23:54.583 INFO:teuthology.orchestra.run.smithi177.stdout:(10/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 4.7 MB 00:03 2024-03-24T00:23:54.783 INFO:teuthology.orchestra.run.smithi177.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.1 MB/s | 220 kB 00:00 2024-03-24T00:23:54.892 INFO:teuthology.orchestra.run.smithi177.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 250 kB/s | 27 kB 00:00 2024-03-24T00:23:55.009 INFO:teuthology.orchestra.run.smithi177.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 268 kB/s | 31 kB 00:00 2024-03-24T00:23:55.206 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.1 MB 00:00 2024-03-24T00:23:55.551 INFO:teuthology.orchestra.run.smithi177.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 1.3 MB/s | 746 kB 00:00 2024-03-24T00:23:55.659 INFO:teuthology.orchestra.run.smithi204.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.1 MB/s | 20 MB 00:05 2024-03-24T00:23:55.735 INFO:teuthology.orchestra.run.smithi204.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-24T00:23:55.743 INFO:teuthology.orchestra.run.smithi177.stdout:(15/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 3.8 MB/s | 20 MB 00:05 2024-03-24T00:23:55.768 INFO:teuthology.orchestra.run.smithi177.stdout:(16/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 752 kB/s | 163 kB 00:00 2024-03-24T00:23:55.793 INFO:teuthology.orchestra.run.smithi204.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 466 kB/s | 27 kB 00:00 2024-03-24T00:23:55.835 INFO:teuthology.orchestra.run.smithi177.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-24T00:23:55.877 INFO:teuthology.orchestra.run.smithi204.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 376 kB/s | 31 kB 00:00 2024-03-24T00:23:56.127 INFO:teuthology.orchestra.run.smithi177.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 1.4 MB/s | 501 kB 00:00 2024-03-24T00:23:56.135 INFO:teuthology.orchestra.run.smithi204.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.8 MB/s | 746 kB 00:00 2024-03-24T00:23:56.210 INFO:teuthology.orchestra.run.smithi177.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 542 kB/s | 45 kB 00:00 2024-03-24T00:23:56.227 INFO:teuthology.orchestra.run.smithi204.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-24T00:23:56.302 INFO:teuthology.orchestra.run.smithi177.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-24T00:23:56.319 INFO:teuthology.orchestra.run.smithi204.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-24T00:23:56.494 INFO:teuthology.orchestra.run.smithi177.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 847 kB/s | 162 kB 00:00 2024-03-24T00:23:56.503 INFO:teuthology.orchestra.run.smithi204.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.7 MB/s | 501 kB 00:00 2024-03-24T00:23:56.778 INFO:teuthology.orchestra.run.smithi177.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.1 MB/s | 321 kB 00:00 2024-03-24T00:23:56.915 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-24T00:23:57.103 INFO:teuthology.orchestra.run.smithi177.stdout:(23/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 4.1 MB/s | 5.1 MB 00:01 2024-03-24T00:23:57.128 INFO:teuthology.orchestra.run.smithi177.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 852 kB/s | 298 kB 00:00 2024-03-24T00:23:57.204 INFO:teuthology.orchestra.run.smithi177.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 993 kB/s | 99 kB 00:00 2024-03-24T00:23:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 786 kB/s | 85 kB 00:00 2024-03-24T00:23:57.446 INFO:teuthology.orchestra.run.smithi177.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 819 kB/s | 170 kB 00:00 2024-03-24T00:23:57.546 INFO:teuthology.orchestra.run.smithi177.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 340 kB/s | 34 kB 00:00 2024-03-24T00:23:57.763 INFO:teuthology.orchestra.run.smithi177.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 640 kB/s | 139 kB 00:00 2024-03-24T00:23:57.820 INFO:teuthology.orchestra.run.smithi204.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.9 MB/s | 5.1 MB 00:01 2024-03-24T00:23:57.903 INFO:teuthology.orchestra.run.smithi204.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 542 kB/s | 45 kB 00:00 2024-03-24T00:23:57.995 INFO:teuthology.orchestra.run.smithi204.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-24T00:23:58.062 INFO:teuthology.orchestra.run.smithi204.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-24T00:23:58.072 INFO:teuthology.orchestra.run.smithi177.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 3.6 MB/s | 3.1 MB 00:00 2024-03-24T00:23:58.246 INFO:teuthology.orchestra.run.smithi204.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 321 kB 00:00 2024-03-24T00:23:58.346 INFO:teuthology.orchestra.run.smithi204.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-24T00:23:58.438 INFO:teuthology.orchestra.run.smithi204.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.1 MB/s | 99 kB 00:00 2024-03-24T00:23:58.523 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:23:58.530 INFO:teuthology.orchestra.run.smithi204.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 932 kB/s | 85 kB 00:00 2024-03-24T00:23:58.948 INFO:teuthology.orchestra.run.smithi177.stdout:(31/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.6 MB/s | 48 MB 00:04 2024-03-24T00:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 1.5 MB/s | 1.8 MB 00:01 2024-03-24T00:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-24T00:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 737 kB/s | 49 kB 00:00 2024-03-24T00:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 245 kB/s | 16 kB 00:00 2024-03-24T00:23:59.166 INFO:teuthology.orchestra.run.smithi177.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 8.7 MB/s | 748 kB 00:00 2024-03-24T00:23:59.200 INFO:teuthology.orchestra.run.smithi177.stdout:(37/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.8 MB/s | 264 kB 00:00 2024-03-24T00:23:59.214 INFO:teuthology.orchestra.run.smithi204.stdout:(26/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.4 MB/s | 48 MB 00:05 2024-03-24T00:23:59.225 INFO:teuthology.orchestra.run.smithi177.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 312 kB/s | 18 kB 00:00 2024-03-24T00:23:59.273 INFO:teuthology.orchestra.run.smithi204.stdout:(27/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.2 MB/s | 3.1 MB 00:00 2024-03-24T00:23:59.298 INFO:teuthology.orchestra.run.smithi204.stdout:(28/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-24T00:23:59.323 INFO:teuthology.orchestra.run.smithi204.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 675 kB/s | 34 kB 00:00 2024-03-24T00:23:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-24T00:23:59.376 INFO:teuthology.orchestra.run.smithi177.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-24T00:23:59.417 INFO:teuthology.orchestra.run.smithi177.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-24T00:23:59.443 INFO:teuthology.orchestra.run.smithi177.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T00:23:59.468 INFO:teuthology.orchestra.run.smithi177.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T00:23:59.493 INFO:teuthology.orchestra.run.smithi177.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-03-24T00:23:59.582 INFO:teuthology.orchestra.run.smithi204.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 6.9 MB/s | 1.8 MB 00:00 2024-03-24T00:23:59.627 INFO:teuthology.orchestra.run.smithi177.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-03-24T00:23:59.649 INFO:teuthology.orchestra.run.smithi204.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-24T00:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-03-24T00:23:59.707 INFO:teuthology.orchestra.run.smithi204.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 848 kB/s | 49 kB 00:00 2024-03-24T00:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 76 kB/s | 17 kB 00:00 2024-03-24T00:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-03-24T00:23:59.766 INFO:teuthology.orchestra.run.smithi204.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 282 kB/s | 16 kB 00:00 2024-03-24T00:23:59.769 INFO:teuthology.orchestra.run.smithi177.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-24T00:23:59.795 INFO:teuthology.orchestra.run.smithi177.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 890 kB/s | 44 kB 00:00 2024-03-24T00:23:59.820 INFO:teuthology.orchestra.run.smithi177.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-24T00:23:59.845 INFO:teuthology.orchestra.run.smithi177.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-24T00:23:59.858 INFO:teuthology.orchestra.run.smithi204.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-24T00:23:59.870 INFO:teuthology.orchestra.run.smithi177.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 623 kB/s | 46 kB 00:00 2024-03-24T00:23:59.892 INFO:teuthology.orchestra.run.smithi204.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.1 MB/s | 264 kB 00:00 2024-03-24T00:23:59.896 INFO:teuthology.orchestra.run.smithi177.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T00:23:59.929 INFO:teuthology.orchestra.run.smithi177.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-03-24T00:23:59.942 INFO:teuthology.orchestra.run.smithi204.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 8.7 MB/s | 748 kB 00:00 2024-03-24T00:23:59.954 INFO:teuthology.orchestra.run.smithi177.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-24T00:23:59.970 INFO:teuthology.orchestra.run.smithi204.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 243 kB/s | 18 kB 00:00 2024-03-24T00:23:59.980 INFO:teuthology.orchestra.run.smithi177.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 409 kB/s | 34 kB 00:00 2024-03-24T00:24:00.126 INFO:teuthology.orchestra.run.smithi204.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-24T00:24:00.138 INFO:teuthology.orchestra.run.smithi177.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-24T00:24:00.168 INFO:teuthology.orchestra.run.smithi204.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-24T00:24:00.180 INFO:teuthology.orchestra.run.smithi177.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 6.7 MB/s | 1.5 MB 00:00 2024-03-24T00:24:00.193 INFO:teuthology.orchestra.run.smithi204.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T00:24:00.206 INFO:teuthology.orchestra.run.smithi177.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.5 MB/s | 811 kB 00:00 2024-03-24T00:24:00.219 INFO:teuthology.orchestra.run.smithi204.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T00:24:00.231 INFO:teuthology.orchestra.run.smithi177.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-24T00:24:00.235 INFO:teuthology.orchestra.run.smithi204.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 763 kB/s | 32 kB 00:00 2024-03-24T00:24:00.281 INFO:teuthology.orchestra.run.smithi177.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-03-24T00:24:00.336 INFO:teuthology.orchestra.run.smithi204.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-24T00:24:00.353 INFO:teuthology.orchestra.run.smithi204.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-24T00:24:00.370 INFO:teuthology.orchestra.run.smithi204.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 300 kB/s | 10 kB 00:00 2024-03-24T00:24:00.381 INFO:teuthology.orchestra.run.smithi177.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-24T00:24:00.395 INFO:teuthology.orchestra.run.smithi204.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-24T00:24:00.406 INFO:teuthology.orchestra.run.smithi177.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 443 kB/s | 78 kB 00:00 2024-03-24T00:24:00.412 INFO:teuthology.orchestra.run.smithi204.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T00:24:00.454 INFO:teuthology.orchestra.run.smithi204.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T00:24:00.479 INFO:teuthology.orchestra.run.smithi204.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-24T00:24:00.497 INFO:teuthology.orchestra.run.smithi204.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-24T00:24:00.499 INFO:teuthology.orchestra.run.smithi177.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-24T00:24:00.515 INFO:teuthology.orchestra.run.smithi177.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 185 kB/s | 24 kB 00:00 2024-03-24T00:24:00.522 INFO:teuthology.orchestra.run.smithi204.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-24T00:24:00.547 INFO:teuthology.orchestra.run.smithi204.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T00:24:00.557 INFO:teuthology.orchestra.run.smithi177.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-24T00:24:00.572 INFO:teuthology.orchestra.run.smithi204.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-24T00:24:00.582 INFO:teuthology.orchestra.run.smithi177.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-24T00:24:00.589 INFO:teuthology.orchestra.run.smithi204.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-03-24T00:24:00.616 INFO:teuthology.orchestra.run.smithi177.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 540 kB/s | 32 kB 00:00 2024-03-24T00:24:00.725 INFO:teuthology.orchestra.run.smithi177.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-24T00:24:00.740 INFO:teuthology.orchestra.run.smithi204.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-24T00:24:00.750 INFO:teuthology.orchestra.run.smithi177.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-24T00:24:00.775 INFO:teuthology.orchestra.run.smithi177.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-24T00:24:00.783 INFO:teuthology.orchestra.run.smithi204.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.9 MB/s | 1.5 MB 00:00 2024-03-24T00:24:00.801 INFO:teuthology.orchestra.run.smithi177.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 918 kB/s | 69 kB 00:00 2024-03-24T00:24:00.817 INFO:teuthology.orchestra.run.smithi204.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-24T00:24:00.826 INFO:teuthology.orchestra.run.smithi177.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-24T00:24:00.842 INFO:teuthology.orchestra.run.smithi204.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T00:24:00.851 INFO:teuthology.orchestra.run.smithi177.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-24T00:24:00.876 INFO:teuthology.orchestra.run.smithi177.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T00:24:00.902 INFO:teuthology.orchestra.run.smithi177.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-24T00:24:00.935 INFO:teuthology.orchestra.run.smithi177.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-24T00:24:00.961 INFO:teuthology.orchestra.run.smithi177.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 511 kB/s | 43 kB 00:00 2024-03-24T00:24:00.994 INFO:teuthology.orchestra.run.smithi204.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-24T00:24:01.027 INFO:teuthology.orchestra.run.smithi204.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-24T00:24:01.052 INFO:teuthology.orchestra.run.smithi204.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T00:24:01.227 INFO:teuthology.orchestra.run.smithi204.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-24T00:24:01.252 INFO:teuthology.orchestra.run.smithi204.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 123 kB/s | 24 kB 00:00 2024-03-24T00:24:01.278 INFO:teuthology.orchestra.run.smithi204.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-24T00:24:01.303 INFO:teuthology.orchestra.run.smithi204.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-24T00:24:01.311 INFO:teuthology.orchestra.run.smithi177.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-24T00:24:01.320 INFO:teuthology.orchestra.run.smithi204.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 758 kB/s | 32 kB 00:00 2024-03-24T00:24:01.336 INFO:teuthology.orchestra.run.smithi177.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 104 kB/s | 39 kB 00:00 2024-03-24T00:24:01.361 INFO:teuthology.orchestra.run.smithi177.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 103 kB/s | 41 kB 00:00 2024-03-24T00:24:01.397 INFO:teuthology.orchestra.run.smithi177.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-24T00:24:01.420 INFO:teuthology.orchestra.run.smithi177.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-24T00:24:01.437 INFO:teuthology.orchestra.run.smithi177.stdout:(84/174): uuid-1.6.2-55.el9.x86_64.rpm 753 kB/s | 56 kB 00:00 2024-03-24T00:24:01.437 INFO:teuthology.orchestra.run.smithi204.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-24T00:24:01.453 INFO:teuthology.orchestra.run.smithi177.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-03-24T00:24:01.462 INFO:teuthology.orchestra.run.smithi204.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-24T00:24:01.487 INFO:teuthology.orchestra.run.smithi177.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-24T00:24:01.488 INFO:teuthology.orchestra.run.smithi204.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-24T00:24:01.505 INFO:teuthology.orchestra.run.smithi204.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-24T00:24:01.521 INFO:teuthology.orchestra.run.smithi177.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-24T00:24:01.522 INFO:teuthology.orchestra.run.smithi204.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-24T00:24:01.547 INFO:teuthology.orchestra.run.smithi204.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-24T00:24:01.554 INFO:teuthology.orchestra.run.smithi177.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-24T00:24:01.572 INFO:teuthology.orchestra.run.smithi204.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T00:24:01.579 INFO:teuthology.orchestra.run.smithi177.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-24T00:24:01.598 INFO:teuthology.orchestra.run.smithi204.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T00:24:01.605 INFO:teuthology.orchestra.run.smithi177.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-24T00:24:01.614 INFO:teuthology.orchestra.run.smithi204.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-24T00:24:01.631 INFO:teuthology.orchestra.run.smithi204.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-24T00:24:01.638 INFO:teuthology.orchestra.run.smithi177.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T00:24:01.740 INFO:teuthology.orchestra.run.smithi204.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 360 kB/s | 39 kB 00:00 2024-03-24T00:24:01.847 INFO:teuthology.orchestra.run.smithi177.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 702 kB/s | 187 kB 00:00 2024-03-24T00:24:01.872 INFO:teuthology.orchestra.run.smithi177.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 838 kB/s | 195 kB 00:00 2024-03-24T00:24:01.906 INFO:teuthology.orchestra.run.smithi177.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T00:24:01.949 INFO:teuthology.orchestra.run.smithi204.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-24T00:24:01.956 INFO:teuthology.orchestra.run.smithi177.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-24T00:24:01.966 INFO:teuthology.orchestra.run.smithi204.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 184 kB/s | 41 kB 00:00 2024-03-24T00:24:01.973 INFO:teuthology.orchestra.run.smithi177.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-03-24T00:24:01.990 INFO:teuthology.orchestra.run.smithi177.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-03-24T00:24:01.991 INFO:teuthology.orchestra.run.smithi204.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-24T00:24:02.009 INFO:teuthology.orchestra.run.smithi204.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-24T00:24:02.025 INFO:teuthology.orchestra.run.smithi204.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T00:24:02.042 INFO:teuthology.orchestra.run.smithi204.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T00:24:02.059 INFO:teuthology.orchestra.run.smithi204.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-24T00:24:02.084 INFO:teuthology.orchestra.run.smithi204.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T00:24:02.118 INFO:teuthology.orchestra.run.smithi204.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T00:24:02.143 INFO:teuthology.orchestra.run.smithi204.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T00:24:02.168 INFO:teuthology.orchestra.run.smithi204.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-24T00:24:02.171 INFO:teuthology.orchestra.run.smithi177.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-03-24T00:24:02.194 INFO:teuthology.orchestra.run.smithi204.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-24T00:24:02.291 INFO:teuthology.orchestra.run.smithi177.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 817 kB/s | 245 kB 00:00 2024-03-24T00:24:02.349 INFO:teuthology.orchestra.run.smithi177.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 600 kB/s | 35 kB 00:00 2024-03-24T00:24:02.399 INFO:teuthology.orchestra.run.smithi177.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 951 kB/s | 216 kB 00:00 2024-03-24T00:24:02.458 INFO:teuthology.orchestra.run.smithi177.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.2 MB/s | 4.1 MB 00:00 2024-03-24T00:24:02.483 INFO:teuthology.orchestra.run.smithi177.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 333 kB/s | 44 kB 00:00 2024-03-24T00:24:02.525 INFO:teuthology.orchestra.run.smithi177.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 748 kB/s | 94 kB 00:00 2024-03-24T00:24:02.625 INFO:teuthology.orchestra.run.smithi177.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-03-24T00:24:02.659 INFO:teuthology.orchestra.run.smithi177.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.0 MB/s | 548 kB 00:00 2024-03-24T00:24:02.701 INFO:teuthology.orchestra.run.smithi177.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 464 kB/s | 19 kB 00:00 2024-03-24T00:24:02.751 INFO:teuthology.orchestra.run.smithi177.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 MB/s | 309 kB 00:00 2024-03-24T00:24:02.753 INFO:teuthology.orchestra.run.smithi204.stdout:(91/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 17 MB 00:09 2024-03-24T00:24:02.778 INFO:teuthology.orchestra.run.smithi204.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 308 kB/s | 187 kB 00:00 2024-03-24T00:24:02.785 INFO:teuthology.orchestra.run.smithi177.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-03-24T00:24:02.810 INFO:teuthology.orchestra.run.smithi177.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-24T00:24:02.886 INFO:teuthology.orchestra.run.smithi177.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-24T00:24:02.886 INFO:teuthology.orchestra.run.smithi204.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-24T00:24:02.911 INFO:teuthology.orchestra.run.smithi177.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-03-24T00:24:02.920 INFO:teuthology.orchestra.run.smithi204.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-24T00:24:02.936 INFO:teuthology.orchestra.run.smithi177.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-24T00:24:02.970 INFO:teuthology.orchestra.run.smithi177.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-24T00:24:02.987 INFO:teuthology.orchestra.run.smithi204.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-03-24T00:24:03.012 INFO:teuthology.orchestra.run.smithi177.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.1 MB/s | 838 kB 00:00 2024-03-24T00:24:03.012 INFO:teuthology.orchestra.run.smithi204.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-24T00:24:03.037 INFO:teuthology.orchestra.run.smithi177.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 651 kB/s | 43 kB 00:00 2024-03-24T00:24:03.062 INFO:teuthology.orchestra.run.smithi177.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-24T00:24:03.087 INFO:teuthology.orchestra.run.smithi177.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-24T00:24:03.104 INFO:teuthology.orchestra.run.smithi204.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-24T00:24:03.112 INFO:teuthology.orchestra.run.smithi177.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-24T00:24:03.138 INFO:teuthology.orchestra.run.smithi177.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-24T00:24:03.155 INFO:teuthology.orchestra.run.smithi204.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-03-24T00:24:03.163 INFO:teuthology.orchestra.run.smithi177.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-24T00:24:03.188 INFO:teuthology.orchestra.run.smithi177.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-03-24T00:24:03.205 INFO:teuthology.orchestra.run.smithi204.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-24T00:24:03.222 INFO:teuthology.orchestra.run.smithi177.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-24T00:24:03.256 INFO:teuthology.orchestra.run.smithi177.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-24T00:24:03.281 INFO:teuthology.orchestra.run.smithi177.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-24T00:24:03.306 INFO:teuthology.orchestra.run.smithi177.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-24T00:24:03.331 INFO:teuthology.orchestra.run.smithi177.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-24T00:24:03.357 INFO:teuthology.orchestra.run.smithi177.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-24T00:24:03.374 INFO:teuthology.orchestra.run.smithi177.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 465 kB/s | 19 kB 00:00 2024-03-24T00:24:03.390 INFO:teuthology.orchestra.run.smithi177.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 591 kB/s | 20 kB 00:00 2024-03-24T00:24:03.397 INFO:teuthology.orchestra.run.smithi204.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-24T00:24:03.422 INFO:teuthology.orchestra.run.smithi204.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 161 kB/s | 35 kB 00:00 2024-03-24T00:24:03.424 INFO:teuthology.orchestra.run.smithi177.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 20 MB/s | 1.0 MB 00:00 2024-03-24T00:24:03.441 INFO:teuthology.orchestra.run.smithi177.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 922 kB/s | 46 kB 00:00 2024-03-24T00:24:03.447 INFO:teuthology.orchestra.run.smithi204.stdout:(102/174): libquadmath-11.4.1-3.el9.x86_64.rpm 281 kB/s | 195 kB 00:00 2024-03-24T00:24:03.464 INFO:teuthology.orchestra.run.smithi204.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 665 kB/s | 44 kB 00:00 2024-03-24T00:24:03.466 INFO:teuthology.orchestra.run.smithi177.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-24T00:24:03.483 INFO:teuthology.orchestra.run.smithi177.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-03-24T00:24:03.489 INFO:teuthology.orchestra.run.smithi204.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-24T00:24:03.517 INFO:teuthology.orchestra.run.smithi177.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-24T00:24:03.540 INFO:teuthology.orchestra.run.smithi204.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-03-24T00:24:03.542 INFO:teuthology.orchestra.run.smithi177.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-24T00:24:03.568 INFO:teuthology.orchestra.run.smithi177.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-24T00:24:03.618 INFO:teuthology.orchestra.run.smithi177.stdout:(138/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.7 MB/s | 2.0 MB 00:00 2024-03-24T00:24:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-24T00:24:03.649 INFO:teuthology.orchestra.run.smithi204.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-03-24T00:24:03.668 INFO:teuthology.orchestra.run.smithi177.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-24T00:24:03.694 INFO:teuthology.orchestra.run.smithi177.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-24T00:24:03.719 INFO:teuthology.orchestra.run.smithi177.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-03-24T00:24:03.744 INFO:teuthology.orchestra.run.smithi177.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-24T00:24:03.769 INFO:teuthology.orchestra.run.smithi177.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-03-24T00:24:03.799 INFO:teuthology.orchestra.run.smithi204.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-24T00:24:03.803 INFO:teuthology.orchestra.run.smithi177.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-24T00:24:03.815 INFO:teuthology.orchestra.run.smithi204.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-03-24T00:24:03.828 INFO:teuthology.orchestra.run.smithi177.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-24T00:24:03.844 INFO:teuthology.orchestra.run.smithi204.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-24T00:24:03.853 INFO:teuthology.orchestra.run.smithi177.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-24T00:24:03.879 INFO:teuthology.orchestra.run.smithi177.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-24T00:24:03.904 INFO:teuthology.orchestra.run.smithi177.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-24T00:24:03.919 INFO:teuthology.orchestra.run.smithi204.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 252 kB/s | 26 kB 00:00 2024-03-24T00:24:03.945 INFO:teuthology.orchestra.run.smithi204.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-24T00:24:03.946 INFO:teuthology.orchestra.run.smithi177.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-24T00:24:03.988 INFO:teuthology.orchestra.run.smithi177.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T00:24:04.053 INFO:teuthology.orchestra.run.smithi204.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-03-24T00:24:04.079 INFO:teuthology.orchestra.run.smithi204.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-24T00:24:04.104 INFO:teuthology.orchestra.run.smithi204.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 952 kB/s | 151 kB 00:00 2024-03-24T00:24:04.137 INFO:teuthology.orchestra.run.smithi204.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-24T00:24:04.171 INFO:teuthology.orchestra.run.smithi204.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-03-24T00:24:04.230 INFO:teuthology.orchestra.run.smithi177.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 9.7 MB/s | 3.4 MB 00:00 2024-03-24T00:24:04.238 INFO:teuthology.orchestra.run.smithi204.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-24T00:24:04.308 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-24T00:24:04.472 INFO:teuthology.orchestra.run.smithi177.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-03-24T00:24:04.563 INFO:teuthology.orchestra.run.smithi204.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 76 kB/s | 32 kB 00:00 2024-03-24T00:24:04.589 INFO:teuthology.orchestra.run.smithi177.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 360 kB/s | 42 kB 00:00 2024-03-24T00:24:04.607 INFO:teuthology.orchestra.run.smithi204.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 34 kB/s | 15 kB 00:00 2024-03-24T00:24:04.633 INFO:teuthology.orchestra.run.smithi204.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 435 kB/s | 172 kB 00:00 2024-03-24T00:24:04.666 INFO:teuthology.orchestra.run.smithi204.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.4 MB/s | 358 kB 00:00 2024-03-24T00:24:04.700 INFO:teuthology.orchestra.run.smithi204.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-24T00:24:04.725 INFO:teuthology.orchestra.run.smithi204.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-24T00:24:04.750 INFO:teuthology.orchestra.run.smithi204.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-24T00:24:04.809 INFO:teuthology.orchestra.run.smithi204.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-24T00:24:04.834 INFO:teuthology.orchestra.run.smithi204.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-24T00:24:04.885 INFO:teuthology.orchestra.run.smithi204.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-24T00:24:04.918 INFO:teuthology.orchestra.run.smithi204.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-24T00:24:04.943 INFO:teuthology.orchestra.run.smithi204.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-24T00:24:04.968 INFO:teuthology.orchestra.run.smithi204.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-03-24T00:24:05.002 INFO:teuthology.orchestra.run.smithi204.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-03-24T00:24:05.103 INFO:teuthology.orchestra.run.smithi204.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 346 kB/s | 46 kB 00:00 2024-03-24T00:24:05.136 INFO:teuthology.orchestra.run.smithi204.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-24T00:24:05.161 INFO:teuthology.orchestra.run.smithi204.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 496 kB/s | 79 kB 00:00 2024-03-24T00:24:05.186 INFO:teuthology.orchestra.run.smithi204.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-03-24T00:24:05.212 INFO:teuthology.orchestra.run.smithi204.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-24T00:24:05.237 INFO:teuthology.orchestra.run.smithi204.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-24T00:24:05.254 INFO:teuthology.orchestra.run.smithi204.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-24T00:24:05.279 INFO:teuthology.orchestra.run.smithi204.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-03-24T00:24:05.305 INFO:teuthology.orchestra.run.smithi204.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-03-24T00:24:05.346 INFO:teuthology.orchestra.run.smithi204.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-24T00:24:05.372 INFO:teuthology.orchestra.run.smithi204.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-24T00:24:05.397 INFO:teuthology.orchestra.run.smithi204.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-03-24T00:24:05.422 INFO:teuthology.orchestra.run.smithi204.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-03-24T00:24:05.448 INFO:teuthology.orchestra.run.smithi204.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T00:24:05.481 INFO:teuthology.orchestra.run.smithi204.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-24T00:24:05.506 INFO:teuthology.orchestra.run.smithi204.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-24T00:24:05.540 INFO:teuthology.orchestra.run.smithi204.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-24T00:24:05.566 INFO:teuthology.orchestra.run.smithi204.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T00:24:05.608 INFO:teuthology.orchestra.run.smithi204.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-24T00:24:05.766 INFO:teuthology.orchestra.run.smithi204.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2024-03-24T00:24:05.931 INFO:teuthology.orchestra.run.smithi177.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 185 kB/s | 248 kB 00:01 2024-03-24T00:24:06.092 INFO:teuthology.orchestra.run.smithi204.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-03-24T00:24:06.234 INFO:teuthology.orchestra.run.smithi204.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 3.4 MB 00:00 2024-03-24T00:24:06.322 INFO:teuthology.orchestra.run.smithi177.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 182 kB/s | 71 kB 00:00 2024-03-24T00:24:06.506 INFO:teuthology.orchestra.run.smithi177.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 199 kB/s | 36 kB 00:00 2024-03-24T00:24:06.559 INFO:teuthology.orchestra.run.smithi204.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.9 MB/s | 2.3 MB 00:00 2024-03-24T00:24:06.576 INFO:teuthology.orchestra.run.smithi204.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 123 kB/s | 42 kB 00:00 2024-03-24T00:24:06.667 INFO:teuthology.orchestra.run.smithi204.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-24T00:24:06.776 INFO:teuthology.orchestra.run.smithi204.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 356 kB/s | 71 kB 00:00 2024-03-24T00:24:06.793 INFO:teuthology.orchestra.run.smithi204.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 291 kB/s | 36 kB 00:00 2024-03-24T00:24:06.840 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:24:06.842 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:24:06.864 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:24:06.865 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:24:06.866 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:24:06.868 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:24:06.870 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:24:06.874 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:24:06.875 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:24:06.876 INFO:teuthology.orchestra.run.smithi204.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.3 MB/s | 1.8 MB 00:00 2024-03-24T00:24:06.902 INFO:teuthology.orchestra.run.smithi204.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-24T00:24:06.961 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-24T00:24:06.977 INFO:teuthology.orchestra.run.smithi204.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 229 kB/s | 23 kB 00:00 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:24:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:24:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:24:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:24:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:24:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:24:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:24:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:24:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:24:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:24:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:24:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:24:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:24:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:24:06.994 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:Install 150 Packages 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 24 Packages 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 224 M 2024-03-24T00:24:06.995 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-24T00:24:07.011 INFO:teuthology.orchestra.run.smithi204.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 998 kB/s | 108 kB 00:00 2024-03-24T00:24:07.023 INFO:teuthology.orchestra.run.smithi177.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 372 kB/s | 192 kB 00:00 2024-03-24T00:24:07.044 INFO:teuthology.orchestra.run.smithi204.stdout:(163/174): elfutils-libelf-0.190-2.el9.x86_64.r 718 kB/s | 192 kB 00:00 2024-03-24T00:24:07.086 INFO:teuthology.orchestra.run.smithi204.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-24T00:24:07.120 INFO:teuthology.orchestra.run.smithi204.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-24T00:24:07.145 INFO:teuthology.orchestra.run.smithi204.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 635 kB/s | 64 kB 00:00 2024-03-24T00:24:07.187 INFO:teuthology.orchestra.run.smithi204.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 285 kB/s | 29 kB 00:00 2024-03-24T00:24:07.238 INFO:teuthology.orchestra.run.smithi204.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-03-24T00:24:07.346 INFO:teuthology.orchestra.run.smithi204.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-24T00:24:07.447 INFO:teuthology.orchestra.run.smithi204.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 311 kB/s | 31 kB 00:00 2024-03-24T00:24:07.488 INFO:teuthology.orchestra.run.smithi204.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.6 MB/s | 2.3 MB 00:00 2024-03-24T00:24:07.572 INFO:teuthology.orchestra.run.smithi204.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-24T00:24:07.597 INFO:teuthology.orchestra.run.smithi204.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 684 kB/s | 74 kB 00:00 2024-03-24T00:24:07.639 INFO:teuthology.orchestra.run.smithi204.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 2.2 MB 00:00 2024-03-24T00:24:07.643 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:24:07.643 INFO:teuthology.orchestra.run.smithi204.stdout:Total 13 MB/s | 224 MB 00:17 2024-03-24T00:24:07.706 INFO:teuthology.orchestra.run.smithi177.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 374 kB/s | 256 kB 00:00 2024-03-24T00:24:07.815 INFO:teuthology.orchestra.run.smithi177.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-03-24T00:24:07.861 INFO:teuthology.orchestra.run.smithi123.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-24T00:24:08.032 INFO:teuthology.orchestra.run.smithi177.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 501 kB/s | 108 kB 00:00 2024-03-24T00:24:08.121 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-24T00:24:08.194 INFO:teuthology.orchestra.run.smithi123.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.6 MB/s | 890 kB 00:00 2024-03-24T00:24:08.268 INFO:teuthology.orchestra.run.smithi123.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.9 MB/s | 145 kB 00:00 2024-03-24T00:24:08.298 INFO:teuthology.orchestra.run.smithi177.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 581 kB/s | 155 kB 00:00 2024-03-24T00:24:08.391 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-24T00:24:08.391 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-24T00:24:08.449 INFO:teuthology.orchestra.run.smithi177.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 905 kB/s | 136 kB 00:00 2024-03-24T00:24:08.533 INFO:teuthology.orchestra.run.smithi177.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 767 kB/s | 64 kB 00:00 2024-03-24T00:24:08.599 INFO:teuthology.orchestra.run.smithi177.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 431 kB/s | 29 kB 00:00 2024-03-24T00:24:08.636 INFO:teuthology.orchestra.run.smithi123.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-03-24T00:24:08.702 INFO:teuthology.orchestra.run.smithi123.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.3 MB/s | 2.3 MB 00:00 2024-03-24T00:24:08.861 INFO:teuthology.orchestra.run.smithi123.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.7 MB/s | 1.5 MB 00:00 2024-03-24T00:24:09.420 INFO:teuthology.orchestra.run.smithi123.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 6.6 MB/s | 4.7 MB 00:00 2024-03-24T00:24:09.816 INFO:teuthology.orchestra.run.smithi177.stdout:(166/174): NetworkManager-libnm-1.47.2-1.el9.x8 336 kB/s | 1.8 MB 00:05 2024-03-24T00:24:10.126 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-24T00:24:10.127 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-24T00:24:10.212 INFO:teuthology.orchestra.run.smithi123.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 7.7 MB/s | 20 MB 00:02 2024-03-24T00:24:10.304 INFO:teuthology.orchestra.run.smithi123.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 278 kB/s | 25 kB 00:00 2024-03-24T00:24:10.662 INFO:teuthology.orchestra.run.smithi123.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 9.4 MB/s | 17 MB 00:01 2024-03-24T00:24:10.837 INFO:teuthology.orchestra.run.smithi123.stdout:(11/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 7.0 MB/s | 9.9 MB 00:01 2024-03-24T00:24:10.871 INFO:teuthology.orchestra.run.smithi123.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 220 kB 00:00 2024-03-24T00:24:10.904 INFO:teuthology.orchestra.run.smithi123.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 411 kB/s | 27 kB 00:00 2024-03-24T00:24:10.946 INFO:teuthology.orchestra.run.smithi123.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 420 kB/s | 31 kB 00:00 2024-03-24T00:24:11.022 INFO:teuthology.orchestra.run.smithi123.stdout:(15/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 6.3 MB/s | 746 kB 00:00 2024-03-24T00:24:11.055 INFO:teuthology.orchestra.run.smithi123.stdout:(16/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 163 kB 00:00 2024-03-24T00:24:11.097 INFO:teuthology.orchestra.run.smithi123.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-24T00:24:11.164 INFO:teuthology.orchestra.run.smithi123.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.5 MB/s | 501 kB 00:00 2024-03-24T00:24:11.248 INFO:teuthology.orchestra.run.smithi123.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 548 kB/s | 45 kB 00:00 2024-03-24T00:24:11.339 INFO:teuthology.orchestra.run.smithi123.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-24T00:24:11.431 INFO:teuthology.orchestra.run.smithi123.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 162 kB 00:00 2024-03-24T00:24:11.532 INFO:teuthology.orchestra.run.smithi123.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 321 kB 00:00 2024-03-24T00:24:11.632 INFO:teuthology.orchestra.run.smithi123.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-24T00:24:11.740 INFO:teuthology.orchestra.run.smithi123.stdout:(24/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 8.0 MB/s | 5.1 MB 00:00 2024-03-24T00:24:11.766 INFO:teuthology.orchestra.run.smithi123.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 749 kB/s | 99 kB 00:00 2024-03-24T00:24:11.808 INFO:teuthology.orchestra.run.smithi123.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-24T00:24:11.900 INFO:teuthology.orchestra.run.smithi123.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T00:24:11.984 INFO:teuthology.orchestra.run.smithi123.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 412 kB/s | 34 kB 00:00 2024-03-24T00:24:12.117 INFO:teuthology.orchestra.run.smithi123.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.8 MB/s | 3.1 MB 00:00 2024-03-24T00:24:12.151 INFO:teuthology.orchestra.run.smithi123.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 836 kB/s | 139 kB 00:00 2024-03-24T00:24:12.376 INFO:teuthology.orchestra.run.smithi123.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 7.0 MB/s | 1.8 MB 00:00 2024-03-24T00:24:12.484 INFO:teuthology.orchestra.run.smithi123.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.3 MB/s | 248 kB 00:00 2024-03-24T00:24:12.577 INFO:teuthology.orchestra.run.smithi123.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 542 kB/s | 49 kB 00:00 2024-03-24T00:24:12.652 INFO:teuthology.orchestra.run.smithi123.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 222 kB/s | 16 kB 00:00 2024-03-24T00:24:12.752 INFO:teuthology.orchestra.run.smithi123.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.6 MB/s | 264 kB 00:00 2024-03-24T00:24:12.869 INFO:teuthology.orchestra.run.smithi123.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-24T00:24:13.069 INFO:teuthology.orchestra.run.smithi123.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 8.1 MB/s | 7.4 MB 00:00 2024-03-24T00:24:13.086 INFO:teuthology.orchestra.run.smithi123.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 84 kB/s | 18 kB 00:00 2024-03-24T00:24:13.132 INFO:teuthology.orchestra.run.smithi177.stdout:(167/174): NetworkManager-1.47.2-1.el9.x86_64.r 258 kB/s | 2.3 MB 00:09 2024-03-24T00:24:13.191 INFO:teuthology.orchestra.run.smithi177.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 270 kB/s | 1.2 MB 00:04 2024-03-24T00:24:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 4.2 MB/s | 920 kB 00:00 2024-03-24T00:24:13.337 INFO:teuthology.orchestra.run.smithi123.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-24T00:24:13.362 INFO:teuthology.orchestra.run.smithi123.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-24T00:24:13.396 INFO:teuthology.orchestra.run.smithi123.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-24T00:24:13.412 INFO:teuthology.orchestra.run.smithi123.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 662 kB/s | 32 kB 00:00 2024-03-24T00:24:13.513 INFO:teuthology.orchestra.run.smithi123.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-24T00:24:13.538 INFO:teuthology.orchestra.run.smithi123.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 144 kB/s | 17 kB 00:00 2024-03-24T00:24:13.596 INFO:teuthology.orchestra.run.smithi123.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 176 kB/s | 10 kB 00:00 2024-03-24T00:24:13.622 INFO:teuthology.orchestra.run.smithi123.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-24T00:24:13.647 INFO:teuthology.orchestra.run.smithi123.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 917 kB/s | 44 kB 00:00 2024-03-24T00:24:13.689 INFO:teuthology.orchestra.run.smithi123.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-24T00:24:13.739 INFO:teuthology.orchestra.run.smithi123.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-24T00:24:13.765 INFO:teuthology.orchestra.run.smithi123.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 665 kB/s | 46 kB 00:00 2024-03-24T00:24:13.798 INFO:teuthology.orchestra.run.smithi123.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-24T00:24:13.832 INFO:teuthology.orchestra.run.smithi123.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-24T00:24:13.865 INFO:teuthology.orchestra.run.smithi123.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-24T00:24:13.891 INFO:teuthology.orchestra.run.smithi123.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 34 kB 00:00 2024-03-24T00:24:14.033 INFO:teuthology.orchestra.run.smithi123.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-24T00:24:14.083 INFO:teuthology.orchestra.run.smithi123.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.6 MB/s | 1.5 MB 00:00 2024-03-24T00:24:14.133 INFO:teuthology.orchestra.run.smithi123.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.0 MB/s | 811 kB 00:00 2024-03-24T00:24:14.184 INFO:teuthology.orchestra.run.smithi123.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-24T00:24:14.326 INFO:teuthology.orchestra.run.smithi123.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-24T00:24:14.359 INFO:teuthology.orchestra.run.smithi123.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.2 MB/s | 1.0 MB 00:00 2024-03-24T00:24:14.401 INFO:teuthology.orchestra.run.smithi123.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-24T00:24:14.535 INFO:teuthology.orchestra.run.smithi123.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-24T00:24:14.551 INFO:teuthology.orchestra.run.smithi123.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 180 kB/s | 24 kB 00:00 2024-03-24T00:24:14.577 INFO:teuthology.orchestra.run.smithi123.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 782 kB/s | 18 kB 00:00 2024-03-24T00:24:14.602 INFO:teuthology.orchestra.run.smithi123.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-24T00:24:14.619 INFO:teuthology.orchestra.run.smithi123.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 794 kB/s | 32 kB 00:00 2024-03-24T00:24:14.761 INFO:teuthology.orchestra.run.smithi123.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-24T00:24:14.766 INFO:teuthology.orchestra.run.smithi177.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 295 kB/s | 465 kB 00:01 2024-03-24T00:24:14.786 INFO:teuthology.orchestra.run.smithi123.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-24T00:24:14.828 INFO:teuthology.orchestra.run.smithi123.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-24T00:24:14.853 INFO:teuthology.orchestra.run.smithi123.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-24T00:24:14.879 INFO:teuthology.orchestra.run.smithi123.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 857 kB/s | 41 kB 00:00 2024-03-24T00:24:14.912 INFO:teuthology.orchestra.run.smithi123.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-24T00:24:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-24T00:24:14.971 INFO:teuthology.orchestra.run.smithi123.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-24T00:24:14.996 INFO:teuthology.orchestra.run.smithi123.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 846 kB/s | 47 kB 00:00 2024-03-24T00:24:15.022 INFO:teuthology.orchestra.run.smithi123.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 875 kB/s | 43 kB 00:00 2024-03-24T00:24:15.033 INFO:teuthology.orchestra.run.smithi177.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 117 kB/s | 31 kB 00:00 2024-03-24T00:24:15.364 INFO:teuthology.orchestra.run.smithi123.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-24T00:24:15.388 INFO:teuthology.orchestra.run.smithi123.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 110 kB/s | 39 kB 00:00 2024-03-24T00:24:15.906 INFO:teuthology.orchestra.run.smithi123.stdout:(80/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 8.5 MB/s | 48 MB 00:05 2024-03-24T00:24:15.939 INFO:teuthology.orchestra.run.smithi123.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 74 kB/s | 41 kB 00:00 2024-03-24T00:24:15.964 INFO:teuthology.orchestra.run.smithi123.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 522 kB/s | 300 kB 00:00 2024-03-24T00:24:15.990 INFO:teuthology.orchestra.run.smithi123.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 708 kB/s | 56 kB 00:00 2024-03-24T00:24:16.015 INFO:teuthology.orchestra.run.smithi123.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-24T00:24:16.040 INFO:teuthology.orchestra.run.smithi123.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 597 kB/s | 44 kB 00:00 2024-03-24T00:24:16.057 INFO:teuthology.orchestra.run.smithi123.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-24T00:24:16.099 INFO:teuthology.orchestra.run.smithi123.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2024-03-24T00:24:16.141 INFO:teuthology.orchestra.run.smithi123.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T00:24:16.166 INFO:teuthology.orchestra.run.smithi123.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 993 kB/s | 103 kB 00:00 2024-03-24T00:24:16.200 INFO:teuthology.orchestra.run.smithi123.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-24T00:24:16.225 INFO:teuthology.orchestra.run.smithi123.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T00:24:16.468 INFO:teuthology.orchestra.run.smithi123.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 809 kB/s | 195 kB 00:00 2024-03-24T00:24:16.501 INFO:teuthology.orchestra.run.smithi123.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 566 kB/s | 187 kB 00:00 2024-03-24T00:24:16.534 INFO:teuthology.orchestra.run.smithi123.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-03-24T00:24:16.559 INFO:teuthology.orchestra.run.smithi123.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 370 kB/s | 21 kB 00:00 2024-03-24T00:24:16.610 INFO:teuthology.orchestra.run.smithi123.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-24T00:24:16.635 INFO:teuthology.orchestra.run.smithi123.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-03-24T00:24:16.677 INFO:teuthology.orchestra.run.smithi123.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-24T00:24:16.710 INFO:teuthology.orchestra.run.smithi123.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-03-24T00:24:16.744 INFO:teuthology.orchestra.run.smithi123.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-24T00:24:16.750 INFO:teuthology.orchestra.run.smithi177.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 318 kB/s | 2.2 MB 00:06 2024-03-24T00:24:16.769 INFO:teuthology.orchestra.run.smithi123.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 602 kB/s | 35 kB 00:00 2024-03-24T00:24:16.803 INFO:teuthology.orchestra.run.smithi123.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-24T00:24:16.825 INFO:teuthology.orchestra.run.smithi177.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 993 kB/s | 74 kB 00:00 2024-03-24T00:24:16.828 INFO:teuthology.orchestra.run.smithi123.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-24T00:24:17.039 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-24T00:24:17.070 INFO:teuthology.orchestra.run.smithi123.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.1 MB/s | 4.1 MB 00:00 2024-03-24T00:24:17.126 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:24:17.169 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:24:17.200 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:17.209 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:17.249 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:24:17.303 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:24:17.337 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:24:17.362 INFO:teuthology.orchestra.run.smithi123.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-24T00:24:17.370 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:24:17.413 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:24:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.6 MB/s | 309 kB 00:00 2024-03-24T00:24:17.479 INFO:teuthology.orchestra.run.smithi123.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 391 kB/s | 19 kB 00:00 2024-03-24T00:24:17.705 INFO:teuthology.orchestra.run.smithi123.stdout:(108/174): lua-devel-5.4.4-4.el9.x86_64.rpm 25 kB/s | 22 kB 00:00 2024-03-24T00:24:17.735 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:24:17.797 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:17.805 INFO:teuthology.orchestra.run.smithi123.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-24T00:24:17.826 INFO:teuthology.orchestra.run.smithi177.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 394 kB/s | 1.1 MB 00:02 2024-03-24T00:24:17.863 INFO:teuthology.orchestra.run.smithi123.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 845 kB/s | 49 kB 00:00 2024-03-24T00:24:17.897 INFO:teuthology.orchestra.run.smithi123.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-03-24T00:24:17.901 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:17.922 INFO:teuthology.orchestra.run.smithi123.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-24T00:24:17.939 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:17.997 INFO:teuthology.orchestra.run.smithi123.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-24T00:24:18.013 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:18.043 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:24:18.056 INFO:teuthology.orchestra.run.smithi123.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 956 kB/s | 151 kB 00:00 2024-03-24T00:24:18.098 INFO:teuthology.orchestra.run.smithi123.stdout:(115/174): protobuf-compiler-3.14.0-13.el9.x86_ 681 kB/s | 863 kB 00:01 2024-03-24T00:24:18.114 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:24:18.123 INFO:teuthology.orchestra.run.smithi123.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 654 kB/s | 43 kB 00:00 2024-03-24T00:24:18.157 INFO:teuthology.orchestra.run.smithi123.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-24T00:24:18.207 INFO:teuthology.orchestra.run.smithi123.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 181 kB/s | 15 kB 00:00 2024-03-24T00:24:18.241 INFO:teuthology.orchestra.run.smithi123.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-24T00:24:18.274 INFO:teuthology.orchestra.run.smithi123.stdout:(120/174): python3-cachetools-4.2.4-1.el9.noarc 184 kB/s | 32 kB 00:00 2024-03-24T00:24:18.285 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:24:18.308 INFO:teuthology.orchestra.run.smithi123.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-24T00:24:18.333 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:24:18.342 INFO:teuthology.orchestra.run.smithi123.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-24T00:24:18.361 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:18.367 INFO:teuthology.orchestra.run.smithi123.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 387 kB/s | 35 kB 00:00 2024-03-24T00:24:18.400 INFO:teuthology.orchestra.run.smithi123.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-24T00:24:18.459 INFO:teuthology.orchestra.run.smithi123.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-03-24T00:24:18.464 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:18.484 INFO:teuthology.orchestra.run.smithi123.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-24T00:24:18.487 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:18.518 INFO:teuthology.orchestra.run.smithi123.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 306 kB/s | 18 kB 00:00 2024-03-24T00:24:18.543 INFO:teuthology.orchestra.run.smithi123.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 399 kB/s | 23 kB 00:00 2024-03-24T00:24:18.551 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:24:18.602 INFO:teuthology.orchestra.run.smithi123.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.9 MB/s | 2.0 MB 00:00 2024-03-24T00:24:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:18.627 INFO:teuthology.orchestra.run.smithi123.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-03-24T00:24:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:18.653 INFO:teuthology.orchestra.run.smithi123.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-24T00:24:18.711 INFO:teuthology.orchestra.run.smithi123.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.5 MB/s | 1.0 MB 00:00 2024-03-24T00:24:18.736 INFO:teuthology.orchestra.run.smithi123.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 561 kB/s | 46 kB 00:00 2024-03-24T00:24:18.781 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:18.785 INFO:teuthology.orchestra.run.smithi123.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 603 kB/s | 79 kB 00:00 2024-03-24T00:24:18.812 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:18.820 INFO:teuthology.orchestra.run.smithi123.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 534 kB/s | 58 kB 00:00 2024-03-24T00:24:18.845 INFO:teuthology.orchestra.run.smithi123.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 275 kB/s | 16 kB 00:00 2024-03-24T00:24:18.862 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:24:18.879 INFO:teuthology.orchestra.run.smithi123.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-24T00:24:18.904 INFO:teuthology.orchestra.run.smithi123.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 487 kB/s | 28 kB 00:00 2024-03-24T00:24:18.929 INFO:teuthology.orchestra.run.smithi123.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2024-03-24T00:24:18.963 INFO:teuthology.orchestra.run.smithi123.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 371 kB/s | 31 kB 00:00 2024-03-24T00:24:18.968 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:24:18.988 INFO:teuthology.orchestra.run.smithi123.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-24T00:24:19.022 INFO:teuthology.orchestra.run.smithi123.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-24T00:24:19.047 INFO:teuthology.orchestra.run.smithi123.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-03-24T00:24:19.071 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:19.072 INFO:teuthology.orchestra.run.smithi123.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 613 kB/s | 51 kB 00:00 2024-03-24T00:24:19.106 INFO:teuthology.orchestra.run.smithi123.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-24T00:24:19.131 INFO:teuthology.orchestra.run.smithi123.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-24T00:24:19.132 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:19.165 INFO:teuthology.orchestra.run.smithi123.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-03-24T00:24:19.184 INFO:teuthology.orchestra.run.smithi177.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 386 kB/s | 2.3 MB 00:06 2024-03-24T00:24:19.188 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:24:19.188 INFO:teuthology.orchestra.run.smithi177.stdout:Total 7.6 MB/s | 224 MB 00:29 2024-03-24T00:24:19.190 INFO:teuthology.orchestra.run.smithi123.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-03-24T00:24:19.217 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:19.224 INFO:teuthology.orchestra.run.smithi123.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-24T00:24:19.266 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:19.290 INFO:teuthology.orchestra.run.smithi123.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-24T00:24:19.362 INFO:teuthology.orchestra.run.smithi123.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.5 MB/s | 321 kB 00:00 2024-03-24T00:24:19.490 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:24:19.673 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-24T00:24:19.683 INFO:teuthology.orchestra.run.smithi123.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 6.9 MB/s | 3.4 MB 00:00 2024-03-24T00:24:19.732 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:24:19.767 INFO:teuthology.orchestra.run.smithi123.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-03-24T00:24:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:24:19.833 INFO:teuthology.orchestra.run.smithi123.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.9 MB/s | 2.3 MB 00:00 2024-03-24T00:24:19.865 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:24:19.892 INFO:teuthology.orchestra.run.smithi123.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 338 kB/s | 42 kB 00:00 2024-03-24T00:24:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:24:19.942 INFO:teuthology.orchestra.run.smithi123.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-24T00:24:19.945 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-24T00:24:19.945 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-24T00:24:19.964 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:24:19.987 INFO:teuthology.orchestra.run.smithi123.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 763 kB/s | 71 kB 00:00 2024-03-24T00:24:20.012 INFO:teuthology.orchestra.run.smithi123.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 531 kB/s | 36 kB 00:00 2024-03-24T00:24:20.021 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:24:20.059 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:24:20.079 INFO:teuthology.orchestra.run.smithi123.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.8 MB/s | 256 kB 00:00 2024-03-24T00:24:20.113 INFO:teuthology.orchestra.run.smithi123.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-24T00:24:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:24:20.138 INFO:teuthology.orchestra.run.smithi123.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 397 kB/s | 23 kB 00:00 2024-03-24T00:24:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:24:20.189 INFO:teuthology.orchestra.run.smithi123.stdout:(162/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.6 MB/s | 1.8 MB 00:00 2024-03-24T00:24:20.214 INFO:teuthology.orchestra.run.smithi123.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-24T00:24:20.218 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:24:20.247 INFO:teuthology.orchestra.run.smithi123.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-24T00:24:20.252 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:24:20.273 INFO:teuthology.orchestra.run.smithi123.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-24T00:24:20.278 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:24:20.298 INFO:teuthology.orchestra.run.smithi123.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-24T00:24:20.323 INFO:teuthology.orchestra.run.smithi123.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 384 kB/s | 29 kB 00:00 2024-03-24T00:24:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:24:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:24:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:24:20.456 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:24:20.465 INFO:teuthology.orchestra.run.smithi123.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 1.2 MB 00:00 2024-03-24T00:24:20.524 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:24:20.565 INFO:teuthology.orchestra.run.smithi123.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 2.2 MB 00:00 2024-03-24T00:24:20.570 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:24:20.632 INFO:teuthology.orchestra.run.smithi123.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.4 MB/s | 2.3 MB 00:00 2024-03-24T00:24:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:24:20.658 INFO:teuthology.orchestra.run.smithi123.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 342 kB/s | 31 kB 00:00 2024-03-24T00:24:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.0 MB/s | 465 kB 00:00 2024-03-24T00:24:20.725 INFO:teuthology.orchestra.run.smithi123.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 74 kB 00:00 2024-03-24T00:24:20.775 INFO:teuthology.orchestra.run.smithi123.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-24T00:24:20.776 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:24:20.783 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:24:20.784 INFO:teuthology.orchestra.run.smithi123.stdout:Total 16 MB/s | 224 MB 00:13 2024-03-24T00:24:20.831 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:24:21.357 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:24:21.435 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:24:21.519 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:24:21.571 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:21.580 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:21.611 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:24:21.660 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:24:21.669 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-24T00:24:21.670 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-24T00:24:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:24:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:24:21.976 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:24:22.126 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:24:22.215 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-24T00:24:22.247 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:24:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:24:22.557 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-24T00:24:22.557 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-24T00:24:22.608 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:24:22.653 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:24:22.709 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:24:22.781 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:24:22.812 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:24:22.848 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:24:22.895 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:24:22.931 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:24:22.978 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:24:23.021 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:24:23.155 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:24:23.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:24:23.231 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:24:23.272 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:24:23.366 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:24:23.406 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:24:23.446 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:24:23.472 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:24:23.496 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:24:23.568 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:24:23.739 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:24:23.847 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:24:23.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:24:23.954 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:24:24.009 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:24:24.194 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:24:24.243 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:24:24.379 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:24:24.416 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:24:24.448 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:24:24.479 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:24:24.506 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:24:24.541 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:24:24.580 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:24:24.612 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:24:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:24:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:24:24.863 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-24T00:24:24.864 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-24T00:24:24.904 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:24:25.028 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:24:25.184 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:24:25.212 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:24:25.311 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:24:25.409 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:24:25.459 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:24:25.491 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:24:25.538 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:24:25.636 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:24:25.693 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:24:26.421 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:24:26.465 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:24:26.498 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:24:26.547 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:24:26.634 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:24:26.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:24:26.870 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:24:26.906 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:24:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:24:27.625 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:24:27.746 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:24:28.501 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-24T00:24:28.604 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:24:28.638 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:24:28.677 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:28.688 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:28.859 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:24:28.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:24:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:24:28.974 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:24:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:24:29.031 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:24:29.065 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:24:29.096 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:24:29.127 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:24:29.155 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:24:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:24:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:29.468 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:24:29.493 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:29.519 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:29.550 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:29.583 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:29.615 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:24:29.640 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:29.651 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:24:29.671 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:24:29.712 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:24:29.900 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:24:29.968 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:24:29.996 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:30.091 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:30.114 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:30.116 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:24:30.138 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:30.161 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:30.201 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:24:30.209 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:30.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:30.231 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:24:30.240 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:30.432 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:30.463 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:30.495 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:24:30.599 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:24:30.708 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:30.843 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:30.894 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:31.104 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:31.207 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:24:31.310 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:24:31.349 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:24:31.375 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:24:31.410 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:24:31.484 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:24:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:24:31.577 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:24:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:24:31.693 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:24:31.736 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:24:31.769 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:24:31.797 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:24:31.866 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:24:31.902 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:24:31.946 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:24:31.974 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:24:32.043 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:24:32.088 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:24:32.170 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:24:32.312 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:24:32.366 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:24:32.435 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:32.470 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:32.478 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-24T00:24:32.613 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:24:32.656 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:24:32.694 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:32.740 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:24:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:24:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:24:32.962 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:24:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:24:33.007 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:24:33.055 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:24:33.056 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:33.106 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:33.114 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:24:33.138 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:33.138 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:24:33.138 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:24:33.138 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:33.154 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:24:33.157 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:24:33.204 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:24:33.242 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:24:33.271 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:24:33.540 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:24:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:24:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:24:33.743 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:24:33.774 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:33.798 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:24:33.851 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:24:33.905 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:24:33.943 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:24:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:24:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:24:34.045 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:24:34.100 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:24:34.181 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:24:34.213 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:24:34.309 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:24:34.331 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:24:34.375 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:24:34.380 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:24:34.405 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:34.417 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:24:34.551 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:24:34.558 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:24:34.589 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:34.598 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:24:34.636 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:24:34.662 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:24:34.678 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:24:34.715 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:24:34.741 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:34.772 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:24:34.811 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:24:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:24:34.851 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:24:34.877 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:24:34.901 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:24:34.981 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:24:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:35.109 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:24:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:24:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:24:35.217 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:24:35.259 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:24:35.307 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:24:35.363 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:24:35.545 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:24:35.566 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:24:35.605 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:24:35.731 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:24:35.739 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:35.770 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:24:35.802 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:24:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:24:35.833 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:24:35.861 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:24:35.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:24:35.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:24:35.983 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:24:36.013 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:24:36.031 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:36.088 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:24:36.108 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:36.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:24:36.156 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:24:36.198 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:24:36.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:24:36.312 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:24:36.394 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:24:36.439 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:24:36.446 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:24:36.525 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:24:36.602 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:24:36.621 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:24:36.671 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:24:36.738 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:24:36.773 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:24:36.803 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:24:36.858 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:24:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:24:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:24:37.081 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:24:37.171 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:24:37.238 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:24:37.302 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:24:37.352 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:24:37.387 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:24:37.391 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:24:37.460 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:24:37.494 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:24:37.511 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:24:37.541 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:24:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:24:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:24:37.621 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:24:37.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:24:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:24:37.794 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:24:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:24:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:24:37.936 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:24:38.022 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:24:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:24:38.312 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:24:38.405 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:24:38.562 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:24:38.673 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:24:39.097 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:24:39.186 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:24:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:24:39.355 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:39.380 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:24:39.419 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:24:39.488 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:24:39.565 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:24:39.615 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:24:39.883 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:24:39.951 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:24:39.982 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:24:40.013 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:24:40.041 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:24:40.275 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:24:40.351 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:24:40.387 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:40.469 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:40.500 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:24:40.545 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:24:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:24:40.693 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:24:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:24:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:24:41.003 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:24:41.030 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:24:41.032 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:41.103 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:41.105 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:24:41.134 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:24:41.199 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:24:41.263 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:24:41.303 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:24:41.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:24:41.420 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:24:41.484 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:24:41.549 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:24:41.747 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:24:41.826 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:24:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:24:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:24:42.071 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:24:42.111 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:24:42.158 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:24:42.194 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:24:42.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:24:42.385 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:24:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:24:42.713 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:24:42.771 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:24:42.833 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:24:42.913 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:24:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:24:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:24:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:24:43.421 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:24:43.455 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:24:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:24:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:24:43.638 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:24:43.706 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:24:43.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:24:43.812 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:24:43.937 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:24:43.983 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:24:43.991 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:44.025 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:44.026 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:24:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:24:44.097 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:44.098 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:24:44.098 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:24:44.098 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:44.126 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:24:44.161 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:24:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:24:44.412 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:24:44.474 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:24:44.510 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:24:44.567 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:24:44.703 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:24:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-03-24T00:24:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-03-24T00:24:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:45.173 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:24:45.189 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:24:45.190 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:45.190 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:24:45.190 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:24:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:24:45.816 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:24:45.898 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:24:46.011 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:24:46.050 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:24:46.282 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:24:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:24:46.399 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:24:46.679 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:24:46.704 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:24:46.704 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:46.705 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:24:46.705 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:24:46.705 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:24:46.705 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:46.929 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:24:46.952 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:24:46.953 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:46.953 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:24:46.953 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:24:46.953 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:24:46.953 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:24:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:24:47.397 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:24:47.420 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:47.523 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:24:47.916 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:24:47.947 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:24:47.979 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:24:48.040 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:24:48.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:24:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:24:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:24:48.838 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:24:48.925 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:24:48.972 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:24:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:24:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:24:49.302 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:24:49.338 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:24:49.415 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:24:49.440 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:24:49.449 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:49.569 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:24:49.600 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:24:49.605 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:24:49.632 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:24:49.655 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:24:49.655 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:49.656 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:24:49.656 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:24:49.656 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:24:49.656 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:49.659 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:24:49.699 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:24:49.710 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:24:49.755 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:24:50.179 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:24:50.202 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:50.276 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:24:50.304 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:24:50.628 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:24:50.660 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:24:50.683 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:50.727 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:24:50.749 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:24:50.749 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:50.750 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:24:50.750 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:50.794 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:24:50.817 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:24:51.144 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:24:52.914 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:52.952 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:53.576 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:24:53.614 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:53.792 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:24:53.792 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:24:53.792 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:24:53.792 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:24:53.839 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:24:55.602 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-03-24T00:24:55.618 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-03-24T00:24:55.619 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-03-24T00:24:55.619 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-03-24T00:24:55.619 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:55.641 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:24:55.671 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:24:55.698 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:24:55.788 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:24:55.807 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:24:55.807 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:55.807 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:24:55.807 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:24:55.998 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:24:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:24:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:24:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:24:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:24:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:24:56.255 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:24:56.285 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:24:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:24:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:24:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:24:57.039 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:24:57.073 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:24:57.093 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:24:57.300 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:24:57.325 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:57.542 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:24:57.564 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:24:57.564 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:57.564 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:24:57.565 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:24:57.565 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:24:57.565 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:57.923 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:24:58.009 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:24:58.032 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:24:58.032 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:24:58.032 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:24:58.032 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:24:58.032 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:24:58.033 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:24:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:24:58.118 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:24:58.128 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:24:58.200 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:24:58.231 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:24:58.256 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:24:58.281 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:24:58.308 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:24:58.308 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:24:58.379 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:24:58.417 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:24:58.417 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:24:58.510 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:24:58.536 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:24:58.570 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:24:58.570 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:24:58.595 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:24:58.601 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:24:58.620 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:24:58.641 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:24:58.653 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:24:58.693 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:24:58.709 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:24:58.725 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:24:58.741 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:24:58.750 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:24:58.775 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:24:58.801 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:24:58.826 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:24:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:24:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:24:58.975 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:24:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:24:59.012 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:24:59.012 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:24:59.104 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:24:59.128 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:24:59.160 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:24:59.188 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:24:59.213 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:24:59.277 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:24:59.968 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:24:59.992 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:25:00.091 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:25:00.246 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:25:00.278 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:25:00.300 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:25:00.300 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:00.301 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:25:00.301 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:25:00.301 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:25:00.301 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:25:01.271 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:25:01.305 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:01.373 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:25:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:25:01.395 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:01.395 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:25:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:01.440 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:01.462 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:01.462 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:01.462 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:25:01.463 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:25:01.463 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:25:01.463 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:25:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:25:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:25:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:25:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:25:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:25:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:25:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:25:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:25:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:25:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:25:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:25:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:25:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:25:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:25:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:25:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:25:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:25:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:25:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:25:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:25:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:25:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:25:06.228 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:25:06.269 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:25:06.299 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:25:06.327 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:25:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:25:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-03-24T00:25:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-03-24T00:25:06.634 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-03-24T00:25:06.634 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-03-24T00:25:06.634 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:06.658 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:25:06.698 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:25:06.758 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:25:06.801 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:25:06.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:25:06.858 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:25:06.860 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:25:06.888 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:25:06.909 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:25:06.909 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:06.910 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:25:06.910 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:06.938 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:25:06.993 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:25:07.036 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:25:07.190 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:25:07.190 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:25:07.190 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:25:07.192 INFO:teuthology.orchestra.run.smithi204.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.193 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:25:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:25:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:25:07.196 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:25:07.197 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:25:07.198 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:25:07.199 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:25:07.200 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:25:07.201 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:25:07.202 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:25:07.203 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:25:07.204 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:25:07.204 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:25:07.204 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-24T00:25:07.655 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:07.674 DEBUG:teuthology.parallel:result is None 2024-03-24T00:25:07.692 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:07.712 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:25:08.531 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:25:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:08.597 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:25:08.637 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:25:08.718 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:25:08.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:25:08.776 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:25:08.801 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:25:08.828 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:25:08.828 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:25:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:25:08.916 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:25:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:25:08.933 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:08.933 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:25:08.933 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:25:08.934 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:25:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:08.945 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:25:08.945 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:25:09.046 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:25:09.073 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:25:09.106 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:25:09.132 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:25:09.156 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:25:09.189 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:25:09.229 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:25:09.261 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:25:09.286 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:25:09.311 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:25:09.337 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:25:09.370 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:25:09.377 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:09.377 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:25:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:25:09.648 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:25:09.659 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:25:09.672 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:25:09.705 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:25:09.732 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:25:09.757 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:25:09.821 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:25:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:25:10.324 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:25:10.377 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:25:10.476 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:25:10.510 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:25:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:25:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:25:11.982 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:25:12.011 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:25:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:25:12.325 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:25:12.355 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:25:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:25:12.437 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:12.437 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:25:12.437 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:25:12.438 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:25:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:25:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:25:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:25:13.585 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:25:13.616 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:25:13.703 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:13.750 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:25:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:25:13.846 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:13.846 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:25:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:14.030 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:14.031 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:14.031 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:25:14.031 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:25:14.031 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:25:14.031 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:14.418 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:25:14.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:14.676 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:25:14.677 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:25:14.678 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:25:14.695 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:25:14.696 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:25:14.697 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:25:14.698 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:25:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:25:14.700 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:25:14.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:25:14.702 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:25:14.703 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:25:14.704 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:25:14.706 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:25:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:25:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-24T00:25:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:25:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:25:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:25:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:25:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:25:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:25:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-24T00:25:17.704 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-24T00:25:18.149 DEBUG:teuthology.parallel:result is None 2024-03-24T00:25:19.349 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:25:19.405 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:25:19.458 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:25:19.503 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:25:19.795 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:25:19.916 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:25:19.964 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:25:20.030 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:25:20.089 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:25:20.133 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:25:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:25:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:25:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:25:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:25:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:25:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:25:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:21.479 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:21.545 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:25:22.891 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:25:22.984 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:25:23.048 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:25:23.164 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:25:23.203 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:25:23.228 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:25:23.262 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:25:23.302 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:25:23.302 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:25:23.424 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:25:23.497 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:25:23.498 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:25:23.629 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:25:23.673 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:25:23.728 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:25:23.760 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:25:23.793 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:25:23.869 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:25:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:25:23.998 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:25:24.031 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:25:24.090 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:25:24.123 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:25:24.156 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:25:24.192 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:24.193 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:24.515 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:25:24.516 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:25:24.671 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:25:24.701 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:25:24.756 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:25:24.787 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:25:24.828 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:25:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:25:25.191 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:25:25.191 INFO:teuthology.orchestra.run.smithi123.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-24T00:25:25.191 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:25:29.956 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:25:29.957 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:25:29.958 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:25:29.958 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:25:29.958 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:25:29.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:25:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:25:29.975 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:25:29.976 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:25:29.977 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:25:29.978 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:25:29.979 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:25:29.979 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:25:29.979 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:25:29.979 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:25:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:25:29.983 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:25:29.984 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:25:29.985 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:25:29.986 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:25:29.987 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:25:32.496 INFO:teuthology.orchestra.run.smithi123.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:32.497 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.498 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.499 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:25:32.500 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.501 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:25:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:25:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:25:32.504 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:25:32.505 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:25:32.506 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.507 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:25:32.508 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-24T00:25:33.208 DEBUG:teuthology.parallel:result is None 2024-03-24T00:25:33.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:33.425 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:25:33.483 INFO:teuthology.orchestra.run.smithi123.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.484 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.484 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:25:33.486 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:33.699 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:25:33.724 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.725 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.725 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:25:33.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:33.858 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:25:33.883 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.883 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:25:33.883 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:25:33.884 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T00:25:33.885 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:25:33.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:25:33.960 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:25:33.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:25:33.995 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:25:33.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:25:34.030 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T00:25:34.030 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:25:34.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:25:34.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:25:34.200 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:25:34.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:25:34.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:25:34.301 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:25:34.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:25:34.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:25:34.397 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T00:25:34.397 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:25:34.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:25:34.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:25:34.582 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:25:34.582 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:25:34.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:25:34.677 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:25:34.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:25:34.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:25:34.772 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T00:25:34.773 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:25:34.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:25:34.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:25:34.969 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:25:34.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:25:34.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:25:35.062 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:25:35.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:25:35.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:25:35.156 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T00:25:35.261 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-24T00:25:35.261 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:35.262 INFO:tasks.cephadm:Cluster fsid is 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:25:35.262 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T00:25:35.262 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.123', 'mon.b': '172.21.15.177', 'mon.c': '172.21.15.204'} 2024-03-24T00:25:35.262 INFO:tasks.cephadm:First mon is mon.a on smithi123 2024-03-24T00:25:35.262 INFO:tasks.cephadm:First mgr is x 2024-03-24T00:25:35.262 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T00:25:35.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-03-24T00:25:35.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-03-24T00:25:35.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-03-24T00:25:35.395 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T00:25:35.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:35.546 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T00:25:35.687 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:35.688 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-24T00:25:35.688 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:36.372 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:25 /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:36.372 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:36.992 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:25 /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:36.993 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:37.450 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:25 /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:37.450 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:37.510 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:37.529 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:25:37.564 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-24T00:25:37.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:25:37.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:25:37.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:25:38.320 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:25:38.388 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:25:39.035 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-24T00:26:04.480 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-24T00:26:11.621 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-03-24T00:26:11.622 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-24T00:26:12.206 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:26:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:26:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:26:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-03-24T00:26:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:26:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:26:12.207 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:26:12.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-03-24T00:26:12.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-03-24T00:26:12.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-03-24T00:26:12.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-03-24T00:26:12.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-03-24T00:26:12.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-03-24T00:26:12.598 INFO:tasks.cephadm:Writing seed config... 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T00:26:12.600 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-24T00:26:12.601 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T00:26:12.602 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-24T00:26:12.603 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T00:26:12.603 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T00:26:12.603 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T00:26:12.603 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-24T00:26:12.603 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:26:12.603 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T00:26:12.658 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 082c21f6-e975-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T00:26:12.659 DEBUG:teuthology.orchestra.run.smithi123:mon.a> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T00:26:12.706 DEBUG:teuthology.orchestra.run.smithi123:mgr.x> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T00:26:12.751 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T00:26:12.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.123 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:26:13.027 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:26:13.027 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', '082c21f6-e975-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.123', '--skip-admin-label'] 2024-03-24T00:26:13.083 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:26:13.085 INFO:teuthology.orchestra.run.smithi123.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T00:26:13.085 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying podman|docker is present... 2024-03-24T00:26:13.132 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:26:13.132 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying lvm2 is present... 2024-03-24T00:26:13.133 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying time synchronization is in place... 2024-03-24T00:26:13.151 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:26:13.152 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:26:13.170 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:26:13.171 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:26:13.192 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout enabled 2024-03-24T00:26:13.211 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout active 2024-03-24T00:26:13.212 INFO:teuthology.orchestra.run.smithi123.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:26:13.212 INFO:teuthology.orchestra.run.smithi123.stdout:Repeating the final host check... 2024-03-24T00:26:13.268 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:26:13.268 INFO:teuthology.orchestra.run.smithi123.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T00:26:13.269 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl is present 2024-03-24T00:26:13.269 INFO:teuthology.orchestra.run.smithi123.stdout:lvcreate is present 2024-03-24T00:26:13.286 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:26:13.287 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:26:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:26:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:26:13.322 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout enabled 2024-03-24T00:26:13.342 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout active 2024-03-24T00:26:13.342 INFO:teuthology.orchestra.run.smithi123.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:26:13.343 INFO:teuthology.orchestra.run.smithi123.stdout:Host looks OK 2024-03-24T00:26:13.343 INFO:teuthology.orchestra.run.smithi123.stdout:Cluster fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:13.343 INFO:teuthology.orchestra.run.smithi123.stdout:Acquiring lock 139792477277632 on /run/cephadm/082c21f6-e975-11ee-95cd-87774f69a715.lock 2024-03-24T00:26:13.344 INFO:teuthology.orchestra.run.smithi123.stdout:Lock 139792477277632 acquired on /run/cephadm/082c21f6-e975-11ee-95cd-87774f69a715.lock 2024-03-24T00:26:13.345 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying IP 172.21.15.123 port 3300 ... 2024-03-24T00:26:13.346 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying IP 172.21.15.123 port 6789 ... 2024-03-24T00:26:13.347 INFO:teuthology.orchestra.run.smithi123.stdout:Base mon IP(s) is [172.21.15.123:3300, 172.21.15.123:6789], mon addrv is [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2024-03-24T00:26:13.360 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.123 metric 100 2024-03-24T00:26:13.360 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.123 metric 100 2024-03-24T00:26:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T00:26:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T00:26:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-24T00:26:13.384 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T00:26:13.384 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T00:26:13.385 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:26:13.385 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T00:26:13.385 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-03-24T00:26:13.385 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:26:13.387 INFO:teuthology.orchestra.run.smithi123.stdout:Mon IP `172.21.15.123` is in CIDR network `172.21.0.0/20` 2024-03-24T00:26:13.387 INFO:teuthology.orchestra.run.smithi123.stdout:Mon IP `172.21.15.123` is in CIDR network `172.21.0.0/20` 2024-03-24T00:26:13.388 INFO:teuthology.orchestra.run.smithi123.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T00:26:13.390 INFO:teuthology.orchestra.run.smithi123.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T00:26:13.392 INFO:teuthology.orchestra.run.smithi123.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:26:14.107 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T00:26:15.275 INFO:teuthology.orchestra.run.smithi123.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:15.276 INFO:teuthology.orchestra.run.smithi123.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:15.276 INFO:teuthology.orchestra.run.smithi123.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T00:26:16.493 INFO:teuthology.orchestra.run.smithi123.stdout:stat: stdout 167 167 2024-03-24T00:26:16.494 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial keys... 2024-03-24T00:26:26.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQAxc/9ldvcgMRAAYcxFiZjCqP/avnKRdahHKA== 2024-03-24T00:26:28.016 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQAzc/9l2wvwExAAYr0SoU+gsAQ6wb10Z5qdoQ== 2024-03-24T00:26:29.512 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQA0c/9lpBL3MRAAMn3NVnzgznM9f9o6aG5dRg== 2024-03-24T00:26:29.515 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial monmap... 2024-03-24T00:26:30.629 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:26:30.629 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool for a [v2:172.21.15.123:3300,v1:172.21.15.123:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:setting min_mon_release = quincy 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: set fsid to 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:26:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:Creating mon... 2024-03-24T00:26:32.029 INFO:teuthology.orchestra.run.smithi123.stdout:create mon.a on 2024-03-24T00:26:32.395 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T00:26:32.623 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T00:26:32.909 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target. 2024-03-24T00:26:32.909 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target. 2024-03-24T00:26:33.320 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a 2024-03-24T00:26:33.320 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to reset failed state of unit ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T00:26:33.602 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target.wants/ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715@.service. 2024-03-24T00:26:34.254 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 podman[31100]: 2024-03-24 00:26:34.000787784 +0000 UTC m=+0.127047910 container create 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:26:34.255 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 podman[31100]: 2024-03-24 00:26:34.254062041 +0000 UTC m=+0.380322139 container init 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T00:26:34.372 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:26:34.373 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-24T00:26:34.388 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:26:34.388 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:26:34.388 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-24T00:26:34.388 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:26:34.389 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mon to start... 2024-03-24T00:26:34.389 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mon... 2024-03-24T00:26:34.536 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 podman[31100]: 2024-03-24 00:26:34.264124442 +0000 UTC m=+0.390384518 container start 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:26:34.537 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 bash[31100]: 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a 2024-03-24T00:26:34.537 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 ceph-mon[31119]: mkfs 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:34.537 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 ceph-mon[31119]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:26:34.537 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:34 smithi123 systemd[1]: Started Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:26:35.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:26:35.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: monmap e1: 1 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:26:35.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: fsmap 2024-03-24T00:26:35.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T00:26:35.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: mgrmap e1: no daemons active 2024-03-24T00:26:35.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:35 smithi123 ceph-mon[31119]: from='client.? 172.21.15.123:0/306641975' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T00:26:35.759 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T00:26:35.760 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout id: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout services: 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.73s) 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout data: 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T00:26:35.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T00:26:35.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.978+0000 7f3bf2600700 1 Processor -- start 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.978+0000 7f3bf2600700 1 -- start start 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.978+0000 7f3bf2600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.978+0000 7f3bf2600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec10a670 con 0x7f3bec109c80 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.979+0000 7f3bf1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.980+0000 7f3bf1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33826/0 (socket says 172.21.15.123:33826) 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.980+0000 7f3bf1200700 1 -- 172.21.15.123:0/780742306 learned_addr learned my addr 172.21.15.123:0/780742306 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.980+0000 7f3bf1200700 1 -- 172.21.15.123:0/780742306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec10ae30 con 0x7f3bec109c80 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.981+0000 7f3bf1200700 1 --2- 172.21.15.123:0/780742306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3be00098a0 tx=0x7f3be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d815ad2b8de99e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.981+0000 7f3beb400700 1 -- 172.21.15.123:0/780742306 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be0004160 con 0x7f3bec109c80 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.981+0000 7f3beb400700 1 -- 172.21.15.123:0/780742306 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be0004300 con 0x7f3bec109c80 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.981+0000 7f3beb400700 1 -- 172.21.15.123:0/780742306 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be00036a0 con 0x7f3bec109c80 2024-03-24T00:26:35.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 -- 172.21.15.123:0/780742306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 msgr2=0x7f3bec10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 --2- 172.21.15.123:0/780742306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3be00098a0 tx=0x7f3be0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 -- 172.21.15.123:0/780742306 shutdown_connections 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 --2- 172.21.15.123:0/780742306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec10a0a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 -- 172.21.15.123:0/780742306 >> 172.21.15.123:0/780742306 conn(0x7f3bec07d060 msgr2=0x7f3bec07d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 -- 172.21.15.123:0/780742306 shutdown_connections 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.982+0000 7f3bf2600700 1 -- 172.21.15.123:0/780742306 wait complete. 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.983+0000 7f3bf2600700 1 Processor -- start 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.983+0000 7f3bf2600700 1 -- start start 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf2600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf2600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec10b360 con 0x7f3bec109c80 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33840/0 (socket says 172.21.15.123:33840) 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf1200700 1 -- 172.21.15.123:0/306641975 learned_addr learned my addr 172.21.15.123:0/306641975 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:35.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.984+0000 7f3bf1200700 1 -- 172.21.15.123:0/306641975 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec19d8b0 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3bf1200700 1 --2- 172.21.15.123:0/306641975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3be0000c00 tx=0x7f3be0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be0003db0 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be0011630 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec19db40 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be0011950 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec1a0830 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3be0011af0 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.985+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3be001c450 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.986+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec069290 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:34.989+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3be0032020 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.068+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f3bec02dbb0 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.068+0000 7f3be9600700 1 -- 172.21.15.123:0/306641975 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+313 (secure 0 0 0) 0x7f3be0004260 con 0x7f3bec109c80 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.070+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 msgr2=0x7f3bec19d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.070+0000 7f3bf2600700 1 --2- 172.21.15.123:0/306641975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3be0000c00 tx=0x7f3be0004750 comp rx=0 tx=0).stop 2024-03-24T00:26:35.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.070+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 shutdown_connections 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.070+0000 7f3bf2600700 1 --2- 172.21.15.123:0/306641975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec109c80 0x7f3bec19d370 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.071+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 >> 172.21.15.123:0/306641975 conn(0x7f3bec07d060 msgr2=0x7f3bec193fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.071+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 shutdown_connections 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:35.071+0000 7f3bf2600700 1 -- 172.21.15.123:0/306641975 wait complete. 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:mon is available 2024-03-24T00:26:35.770 INFO:teuthology.orchestra.run.smithi123.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T00:26:36.819 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:36 smithi123 ceph-mon[31119]: from='client.? 172.21.15.123:0/3525863346' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:26:36.819 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:36 smithi123 ceph-mon[31119]: from='client.? 172.21.15.123:0/3525863346' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout fsid = 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:26:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.371+0000 7f73c7000700 1 Processor -- start 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.371+0000 7f73c7000700 1 -- start start 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.372+0000 7f73c7000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.372+0000 7f73c7000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c007c800 con 0x7f73c007dd50 2024-03-24T00:26:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.372+0000 7f73bfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.372+0000 7f73bfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54770/0 (socket says 172.21.15.123:54770) 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.372+0000 7f73bfe00700 1 -- 172.21.15.123:0/232262192 learned_addr learned my addr 172.21.15.123:0/232262192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.373+0000 7f73bfe00700 1 -- 172.21.15.123:0/232262192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c007c980 con 0x7f73c007dd50 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.373+0000 7f73bfe00700 1 --2- 172.21.15.123:0/232262192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f73b00098a0 tx=0x7f73b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3291cb0139072a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.373+0000 7f73bea00700 1 -- 172.21.15.123:0/232262192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b0004160 con 0x7f73c007dd50 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.373+0000 7f73bea00700 1 -- 172.21.15.123:0/232262192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0004300 con 0x7f73c007dd50 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.373+0000 7f73bea00700 1 -- 172.21.15.123:0/232262192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b00036a0 con 0x7f73c007dd50 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 -- 172.21.15.123:0/232262192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 msgr2=0x7f73c007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 --2- 172.21.15.123:0/232262192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f73b00098a0 tx=0x7f73b0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 -- 172.21.15.123:0/232262192 shutdown_connections 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 --2- 172.21.15.123:0/232262192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c007c2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 -- 172.21.15.123:0/232262192 >> 172.21.15.123:0/232262192 conn(0x7f73c01032c0 msgr2=0x7f73c01056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 -- 172.21.15.123:0/232262192 shutdown_connections 2024-03-24T00:26:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.374+0000 7f73c7000700 1 -- 172.21.15.123:0/232262192 wait complete. 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.375+0000 7f73c7000700 1 Processor -- start 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73c7000700 1 -- start start 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73c7000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73c7000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c007ce50 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73bfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73bfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54772/0 (socket says 172.21.15.123:54772) 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.376+0000 7f73bfe00700 1 -- 172.21.15.123:0/3525863346 learned_addr learned my addr 172.21.15.123:0/3525863346 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bfe00700 1 -- 172.21.15.123:0/3525863346 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c019da20 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bfe00700 1 --2- 172.21.15.123:0/3525863346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f73b0000c00 tx=0x7f73b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b0003db0 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0011630 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c019dcb0 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b0011950 con 0x7f73c007dd50 2024-03-24T00:26:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c01a09a0 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.377+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f73b0011af0 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.378+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f73b001c450 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.378+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c0069290 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.381+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f73b0032020 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.457+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f73c002dbb0 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.466+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+483 (secure 0 0 0) 0x7f73b0004260 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.467+0000 7f73bcc00700 1 -- 172.21.15.123:0/3525863346 <== mon.0 v2:172.21.15.123:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73b001b450 con 0x7f73c007dd50 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 msgr2=0x7f73c019d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 --2- 172.21.15.123:0/3525863346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f73b0000c00 tx=0x7f73b0004750 comp rx=0 tx=0).stop 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 shutdown_connections 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 --2- 172.21.15.123:0/3525863346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73c007dd50 0x7f73c019d4e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 >> 172.21.15.123:0/3525863346 conn(0x7f73c01032c0 msgr2=0x7f73c01055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 shutdown_connections 2024-03-24T00:26:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:36.468+0000 7f73c7000700 1 -- 172.21.15.123:0/3525863346 wait complete. 2024-03-24T00:26:37.178 INFO:teuthology.orchestra.run.smithi123.stdout:Generating new minimal ceph.conf... 2024-03-24T00:26:38.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:37 smithi123 ceph-mon[31119]: from='client.? 172.21.15.123:0/3182413626' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:38.535 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.778+0000 7f5844600700 1 Processor -- start 2024-03-24T00:26:38.535 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.778+0000 7f5844600700 1 -- start start 2024-03-24T00:26:38.535 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5844600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:38.535 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5844600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c10a7c0 con 0x7f583c109e60 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5841c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5841c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54786/0 (socket says 172.21.15.123:54786) 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5841c00700 1 -- 172.21.15.123:0/3175355034 learned_addr learned my addr 172.21.15.123:0/3175355034 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.779+0000 7f5841c00700 1 -- 172.21.15.123:0/3175355034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c10af70 con 0x7f583c109e60 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.780+0000 7f5841c00700 1 --2- 172.21.15.123:0/3175355034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3808a02aa021a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.780+0000 7f583be00700 1 -- 172.21.15.123:0/3175355034 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830004160 con 0x7f583c109e60 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.780+0000 7f583be00700 1 -- 172.21.15.123:0/3175355034 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5830004300 con 0x7f583c109e60 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.780+0000 7f583be00700 1 -- 172.21.15.123:0/3175355034 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830011b50 con 0x7f583c109e60 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 -- 172.21.15.123:0/3175355034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 msgr2=0x7f583c10a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 --2- 172.21.15.123:0/3175355034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 -- 172.21.15.123:0/3175355034 shutdown_connections 2024-03-24T00:26:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 --2- 172.21.15.123:0/3175355034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c10a280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 -- 172.21.15.123:0/3175355034 >> 172.21.15.123:0/3175355034 conn(0x7f583c07d2e0 msgr2=0x7f583c07d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 -- 172.21.15.123:0/3175355034 shutdown_connections 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.781+0000 7f5844600700 1 -- 172.21.15.123:0/3175355034 wait complete. 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5844600700 1 Processor -- start 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5844600700 1 -- start start 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5844600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5844600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c10e3b0 con 0x7f583c109e60 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5841c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5841c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54794/0 (socket says 172.21.15.123:54794) 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.782+0000 7f5841c00700 1 -- 172.21.15.123:0/3182413626 learned_addr learned my addr 172.21.15.123:0/3182413626 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f5841c00700 1 -- 172.21.15.123:0/3182413626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c19e360 con 0x7f583c109e60 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f5841c00700 1 --2- 172.21.15.123:0/3182413626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5830011b60 tx=0x7f5830011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830003970 con 0x7f583c109e60 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5830003b10 con 0x7f583c109e60 2024-03-24T00:26:38.537 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830024610 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c19e5f0 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.783+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1a12e0 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.784+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5830024c50 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.784+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5830011e00 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.785+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c069290 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.788+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f583001a9f0 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.864+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f583c1a14b0 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.864+0000 7f583a000700 1 -- 172.21.15.123:0/3182413626 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f583001a430 con 0x7f583c109e60 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.865+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 msgr2=0x7f583c19de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.865+0000 7f5844600700 1 --2- 172.21.15.123:0/3182413626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5830011b60 tx=0x7f5830011c40 comp rx=0 tx=0).stop 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.866+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 shutdown_connections 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.866+0000 7f5844600700 1 --2- 172.21.15.123:0/3182413626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c109e60 0x7f583c19de20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:38.538 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.866+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 >> 172.21.15.123:0/3182413626 conn(0x7f583c07d2e0 msgr2=0x7f583c107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:38.539 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.866+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 shutdown_connections 2024-03-24T00:26:38.539 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:37.866+0000 7f5844600700 1 -- 172.21.15.123:0/3182413626 wait complete. 2024-03-24T00:26:38.539 INFO:teuthology.orchestra.run.smithi123.stdout:Restarting the monitor... 2024-03-24T00:26:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:38 smithi123 systemd[1]: Stopping Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:26:39.295 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a[31115]: 2024-03-24T00:26:38.979+0000 7f2ef4e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:26:39.295 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a[31115]: 2024-03-24T00:26:38.979+0000 7f2ef4e00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T00:26:39.295 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:39 smithi123 podman[31361]: 2024-03-24 00:26:39.063269003 +0000 UTC m=+0.187463606 container died 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:26:39.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:39 smithi123 podman[31361]: 2024-03-24 00:26:39.398946961 +0000 UTC m=+0.523141574 container cleanup 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:26:39.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:39 smithi123 bash[31361]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a 2024-03-24T00:26:40.039 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:39 smithi123 podman[31375]: 2024-03-24 00:26:39.650071758 +0000 UTC m=+0.571778270 container remove 17a62d4cfe1c8aab41a49bbdb7fcc8b891a48f95bd3c5ddfadddd7b51f897b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-03-24T00:26:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T00:26:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 systemd[1]: Stopped Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:26:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service: Consumed 1.784s CPU time. 2024-03-24T00:26:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 systemd[1]: Starting Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:26:40.733 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 podman[31468]: 2024-03-24 00:26:40.442426804 +0000 UTC m=+0.120934417 container create 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:26:40.733 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 podman[31468]: 2024-03-24 00:26:40.355203583 +0000 UTC m=+0.033711160 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:26:40.733 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 podman[31468]: 2024-03-24 00:26:40.678878718 +0000 UTC m=+0.357386357 container init 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:26:40.733 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 podman[31468]: 2024-03-24 00:26:40.688770547 +0000 UTC m=+0.367278156 container start 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T00:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: pidfile_write: ignore empty --pid-file 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: load: jerasure load: lrc 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Git sha 0 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: DB SUMMARY 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: DB Session ID: P7P4I5H05NJX7Q3NVSCX 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:26:41.014 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100420 ; 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.env: 0x55b93103cb20 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.info_log: 0x55b932833960 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.statistics: (nil) 2024-03-24T00:26:41.015 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.use_fsync: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.db_log_dir: 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.wal_dir: 2024-03-24T00:26:41.016 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:26:41.017 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.write_buffer_manager: 0x55b9328061e0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.unordered_write: 0 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:26:41.018 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.row_cache: None 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.wal_filter: None 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.wal_compression: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:26:41.019 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:26:41.020 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_open_files: -1 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Compression algorithms supported: 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kZSTD supported: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:26:41.021 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.merge_operator: 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_filter: None 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b932833a60) 2024-03-24T00:26:41.022 INFO:journalctl@ceph.mon.a.smithi123.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: index_type: 0 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: data_block_index_type: 0 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: index_shortening: 1 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: checksum: 4 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: no_block_cache: 0 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_cache: 0x55b932839090 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_cache_options: 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: capacity : 536870912 2024-03-24T00:26:41.023 INFO:journalctl@ceph.mon.a.smithi123.stdout: num_shard_bits : 4 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: strict_capacity_limit : 0 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_cache_compressed: (nil) 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: persistent_cache: (nil) 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_size: 4096 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_size_deviation: 10 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_restart_interval: 16 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: index_block_restart_interval: 1 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: metadata_block_size: 4096 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: partition_filters: 0 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: use_delta_encoding: 1 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: filter_policy: bloomfilter 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: whole_key_filtering: 1 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: verify_compression: 0 2024-03-24T00:26:41.024 INFO:journalctl@ceph.mon.a.smithi123.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: format_version: 5 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: enable_index_compression: 1 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: block_align: 0 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: max_auto_readahead_size: 262144 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: prepopulate_block_cache: 0 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression: NoCompression 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.num_levels: 7 2024-03-24T00:26:41.025 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:26:41.026 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:26:41.027 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:26:41.028 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:26:41.029 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:26:41.030 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.ttl: 2592000 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:26:41.031 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 318ce286-1d63-4b7b-bb29-291289213ebb 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240000743975, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240000745832, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95589, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93588, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13880, "raw_average_key_size": 50, "raw_value_size": 86011, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711240000, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "318ce286-1d63-4b7b-bb29-291289213ebb", "db_session_id": "P7P4I5H05NJX7Q3NVSCX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240000745983, "job": 1, "event": "recovery_finished"} 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b93292c000 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: DB pointer 0x55b932916000 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:26:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: ** DB Stats ** 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: L0 2/0 95.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:26:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout: Sum 2/0 95.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:26:41.034 INFO:journalctl@ceph.mon.a.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: Cumulative compaction: 0.00 GB write, 14.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: Interval compaction: 0.00 GB write, 14.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: Block cache BinnedLRUCache@0x55b932839090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: starting mon.a rank 0 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???) e1 preinit fsid 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).mds e1 new map 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).mds e1 print_map 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: e1 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi123.stdout: legacy client fscid: -1 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout: 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout: No filesystems configured 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 bash[31468]: 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 systemd[1]: Started Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: monmap e1: 1 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: fsmap 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:40 smithi123 ceph-mon[31487]: mgrmap e1: no daemons active 2024-03-24T00:26:42.378 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.614+0000 7f6a3b600700 1 Processor -- start 2024-03-24T00:26:42.378 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.615+0000 7f6a3b600700 1 -- start start 2024-03-24T00:26:42.378 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.615+0000 7f6a3b600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:42.378 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.615+0000 7f6a3b600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3407be90 con 0x7f6a341086b0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.616+0000 7f6a38c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.616+0000 7f6a38c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54808/0 (socket says 172.21.15.123:54808) 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.616+0000 7f6a38c00700 1 -- 172.21.15.123:0/3014121881 learned_addr learned my addr 172.21.15.123:0/3014121881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.616+0000 7f6a38c00700 1 -- 172.21.15.123:0/3014121881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3410afe0 con 0x7f6a341086b0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.617+0000 7f6a38c00700 1 --2- 172.21.15.123:0/3014121881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a280098a0 tx=0x7f6a28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d17acf6665b786e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.617+0000 7f6a33400700 1 -- 172.21.15.123:0/3014121881 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a28004160 con 0x7f6a341086b0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.617+0000 7f6a33400700 1 -- 172.21.15.123:0/3014121881 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a28004300 con 0x7f6a341086b0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.617+0000 7f6a33400700 1 -- 172.21.15.123:0/3014121881 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a28011b50 con 0x7f6a341086b0 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 -- 172.21.15.123:0/3014121881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 msgr2=0x7f6a3410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 --2- 172.21.15.123:0/3014121881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a280098a0 tx=0x7f6a28009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 -- 172.21.15.123:0/3014121881 shutdown_connections 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 --2- 172.21.15.123:0/3014121881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3410aaa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 -- 172.21.15.123:0/3014121881 >> 172.21.15.123:0/3014121881 conn(0x7f6a34102100 msgr2=0x7f6a34104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:42.379 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 -- 172.21.15.123:0/3014121881 shutdown_connections 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.618+0000 7f6a3b600700 1 -- 172.21.15.123:0/3014121881 wait complete. 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.619+0000 7f6a3b600700 1 Processor -- start 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.620+0000 7f6a3b600700 1 -- start start 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.620+0000 7f6a3b600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.620+0000 7f6a3b600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3410c010 con 0x7f6a341086b0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.621+0000 7f6a38c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.621+0000 7f6a38c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54824/0 (socket says 172.21.15.123:54824) 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.621+0000 7f6a38c00700 1 -- 172.21.15.123:0/1634854904 learned_addr learned my addr 172.21.15.123:0/1634854904 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.621+0000 7f6a38c00700 1 -- 172.21.15.123:0/1634854904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3419c150 con 0x7f6a341086b0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.621+0000 7f6a38c00700 1 --2- 172.21.15.123:0/1634854904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a28011b60 tx=0x7f6a28011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a28003b20 con 0x7f6a341086b0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3419c3e0 con 0x7f6a341086b0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a28003cc0 con 0x7f6a341086b0 2024-03-24T00:26:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a28024840 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3419f0d0 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6a28022020 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.622+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6a28011e00 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.623+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a340637c0 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.629+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6a28043dd0 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.717+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6a3419f2a0 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.720+0000 7f6a31600700 1 -- 172.21.15.123:0/1634854904 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6a28032020 con 0x7f6a341086b0 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.721+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 msgr2=0x7f6a3419bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.721+0000 7f6a3b600700 1 --2- 172.21.15.123:0/1634854904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a28011b60 tx=0x7f6a28011c40 comp rx=0 tx=0).stop 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.722+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 shutdown_connections 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.722+0000 7f6a3b600700 1 --2- 172.21.15.123:0/1634854904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a341086b0 0x7f6a3419bc10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.722+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 >> 172.21.15.123:0/1634854904 conn(0x7f6a34102100 msgr2=0x7f6a34102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.722+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 shutdown_connections 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:41.722+0000 7f6a3b600700 1 -- 172.21.15.123:0/1634854904 wait complete. 2024-03-24T00:26:42.381 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T00:26:42.392 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:26:42.392 INFO:teuthology.orchestra.run.smithi123.stdout:Creating mgr... 2024-03-24T00:26:42.393 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T00:26:42.394 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T00:26:42.800 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x 2024-03-24T00:26:42.801 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to reset failed state of unit ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T00:26:43.050 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1634854904' entity='client.admin' 2024-03-24T00:26:43.067 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715.target.wants/ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-082c21f6-e975-11ee-95cd-87774f69a715@.service. 2024-03-24T00:26:43.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 systemd[1]: Starting Ceph mgr.x for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:26:43.700 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 podman[31714]: 2024-03-24 00:26:43.354223282 +0000 UTC m=+0.034505517 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:26:43.700 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 podman[31714]: 2024-03-24 00:26:43.469731068 +0000 UTC m=+0.150013336 container create 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:26:43.700 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 podman[31714]: 2024-03-24 00:26:43.689729976 +0000 UTC m=+0.370012291 container init 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:26:43.700 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 podman[31714]: 2024-03-24 00:26:43.69892681 +0000 UTC m=+0.379209082 container start 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:26:43.808 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:26:43.808 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-24T00:26:43.826 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:26:43.826 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:26:43.826 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-24T00:26:43.826 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:26:43.849 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:26:43.849 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-24T00:26:43.870 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:26:43.870 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:26:43.871 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-24T00:26:43.871 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T00:26:43.871 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr to start... 2024-03-24T00:26:43.871 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr... 2024-03-24T00:26:44.024 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 bash[31714]: 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 2024-03-24T00:26:44.024 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:43 smithi123 systemd[1]: Started Ceph mgr.x for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:26:44.525 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:44.477+0000 7f791f1f4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:26:44.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:44 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2753218275' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:26:44.803 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:44.629+0000 7f791f1f4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:26:45.233 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:45.233 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:26:45.233 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:26:45.233 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-24T00:26:45.234 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:26:45.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:26:45.236 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:26:34.341914+0000", 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:26:45.237 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.429+0000 7ff1d2200700 1 Processor -- start 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.429+0000 7ff1d2200700 1 -- start start 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.429+0000 7ff1d2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.429+0000 7ff1d2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc07c800 con 0x7ff1cc07dd50 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.430+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.430+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54856/0 (socket says 172.21.15.123:54856) 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.430+0000 7ff1cb400700 1 -- 172.21.15.123:0/209384282 learned_addr learned my addr 172.21.15.123:0/209384282 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.431+0000 7ff1cb400700 1 -- 172.21.15.123:0/209384282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc07c980 con 0x7ff1cc07dd50 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.431+0000 7ff1cb400700 1 --2- 172.21.15.123:0/209384282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009e80 tx=0x7ff1bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b6242973291b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.431+0000 7ff1ca000700 1 -- 172.21.15.123:0/209384282 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc017070 con 0x7ff1cc07dd50 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.431+0000 7ff1ca000700 1 -- 172.21.15.123:0/209384282 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1bc00b8c0 con 0x7ff1cc07dd50 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.431+0000 7ff1ca000700 1 -- 172.21.15.123:0/209384282 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc0045c0 con 0x7ff1cc07dd50 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.432+0000 7ff1d2200700 1 -- 172.21.15.123:0/209384282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 msgr2=0x7ff1cc07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:45.238 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.432+0000 7ff1d2200700 1 --2- 172.21.15.123:0/209384282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009e80 tx=0x7ff1bc00b330 comp rx=0 tx=0).stop 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.432+0000 7ff1d2200700 1 -- 172.21.15.123:0/209384282 shutdown_connections 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.432+0000 7ff1d2200700 1 --2- 172.21.15.123:0/209384282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc07c2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.432+0000 7ff1d2200700 1 -- 172.21.15.123:0/209384282 >> 172.21.15.123:0/209384282 conn(0x7ff1cc1032c0 msgr2=0x7ff1cc1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.433+0000 7ff1d2200700 1 -- 172.21.15.123:0/209384282 shutdown_connections 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.433+0000 7ff1d2200700 1 -- 172.21.15.123:0/209384282 wait complete. 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.434+0000 7ff1d2200700 1 Processor -- start 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.434+0000 7ff1d2200700 1 -- start start 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.434+0000 7ff1d2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.434+0000 7ff1d2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc10e150 con 0x7ff1cc07dd50 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.435+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.435+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54870/0 (socket says 172.21.15.123:54870) 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.435+0000 7ff1cb400700 1 -- 172.21.15.123:0/2753218275 learned_addr learned my addr 172.21.15.123:0/2753218275 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.435+0000 7ff1cb400700 1 -- 172.21.15.123:0/2753218275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc19e2e0 con 0x7ff1cc07dd50 2024-03-24T00:26:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.435+0000 7ff1cb400700 1 --2- 172.21.15.123:0/2753218275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009530 tx=0x7ff1bc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc020020 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1bc003b50 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc19e570 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc017040 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1a1220 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff1bc023020 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.436+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff1bc00f050 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.437+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc069290 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.440+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff1bc02a020 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.519+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff1cc1a17e0 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.519+0000 7ff1c3e00700 1 -- 172.21.15.123:0/2753218275 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7ff1bc00b860 con 0x7ff1cc07dd50 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.520+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 msgr2=0x7ff1cc19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.520+0000 7ff1d2200700 1 --2- 172.21.15.123:0/2753218275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009530 tx=0x7ff1bc00bd50 comp rx=0 tx=0).stop 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.521+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 shutdown_connections 2024-03-24T00:26:45.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.521+0000 7ff1d2200700 1 --2- 172.21.15.123:0/2753218275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1cc07dd50 0x7ff1cc19dda0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:45.241 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.521+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 >> 172.21.15.123:0/2753218275 conn(0x7ff1cc1032c0 msgr2=0x7ff1cc103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:45.241 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.521+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 shutdown_connections 2024-03-24T00:26:45.241 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:44.521+0000 7ff1d2200700 1 -- 172.21.15.123:0/2753218275 wait complete. 2024-03-24T00:26:45.241 INFO:teuthology.orchestra.run.smithi123.stdout:mgr not available, waiting (1/15)... 2024-03-24T00:26:45.564 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:45.444+0000 7f791f1f4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:26:45.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:45.564+0000 7f791f1f4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:26:45.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:45.738+0000 7f791f1f4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:26:46.295 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:45.859+0000 7f791f1f4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:26:46.295 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:45.967+0000 7f791f1f4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:26:46.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:46.295+0000 7f791f1f4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:26:46.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:46.434+0000 7f791f1f4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:26:46.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:46.545+0000 7f791f1f4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:26:47.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:46.667+0000 7f791f1f4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:26:47.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:46.881+0000 7f791f1f4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:26:48.205 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1912976629' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:26:48.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:48.204+0000 7f791f1f4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:26:48.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:48.375+0000 7f791f1f4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:26:48.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:48.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:26:48.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:26:48.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:26:48.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:26:48.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:26:48.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:26:48.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:26:48.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:48.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:26:48.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:48.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:26:48.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:26:48.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:26:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:26:48.652 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:26:48.652 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:26:48.653 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:26:48.659 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:26:34.341914+0000", 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.842+0000 7f5685c00700 1 Processor -- start 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.842+0000 7f5685c00700 1 -- start start 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.843+0000 7f5685c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.843+0000 7f5685c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56801084a0 con 0x7f5680107ab0 2024-03-24T00:26:48.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.843+0000 7f567fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.843+0000 7f567fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35486/0 (socket says 172.21.15.123:35486) 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.843+0000 7f567fe00700 1 -- 172.21.15.123:0/2622701195 learned_addr learned my addr 172.21.15.123:0/2622701195 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.844+0000 7f567fe00700 1 -- 172.21.15.123:0/2622701195 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5680108c60 con 0x7f5680107ab0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.844+0000 7f567fe00700 1 --2- 172.21.15.123:0/2622701195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f56700098a0 tx=0x7f5670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72114475cf442cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.844+0000 7f567ea00700 1 -- 172.21.15.123:0/2622701195 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5670004160 con 0x7f5680107ab0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.844+0000 7f567ea00700 1 -- 172.21.15.123:0/2622701195 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5670004300 con 0x7f5680107ab0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.844+0000 7f567ea00700 1 -- 172.21.15.123:0/2622701195 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5670011b50 con 0x7f5680107ab0 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 -- 172.21.15.123:0/2622701195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 msgr2=0x7f5680107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 --2- 172.21.15.123:0/2622701195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f56700098a0 tx=0x7f5670009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 -- 172.21.15.123:0/2622701195 shutdown_connections 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 --2- 172.21.15.123:0/2622701195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f5680107ed0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 -- 172.21.15.123:0/2622701195 >> 172.21.15.123:0/2622701195 conn(0x7f5680103110 msgr2=0x7f5680105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 -- 172.21.15.123:0/2622701195 shutdown_connections 2024-03-24T00:26:48.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.845+0000 7f5685c00700 1 -- 172.21.15.123:0/2622701195 wait complete. 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.846+0000 7f5685c00700 1 Processor -- start 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f5685c00700 1 -- start start 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f5685c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f5685c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568010c010 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f567fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f567fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35496/0 (socket says 172.21.15.123:35496) 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.847+0000 7f567fe00700 1 -- 172.21.15.123:0/1912976629 learned_addr learned my addr 172.21.15.123:0/1912976629 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f567fe00700 1 -- 172.21.15.123:0/1912976629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568019e1f0 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f567fe00700 1 --2- 172.21.15.123:0/1912976629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5670011850 tx=0x7f5670011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5670003af0 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5670020070 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568019e480 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56700195a0 con 0x7f5680107ab0 2024-03-24T00:26:48.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.848+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56801a1170 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.849+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5670019be0 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.849+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5670011e00 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.849+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680069290 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.854+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5670003c90 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.932+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f56801a1730 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.933+0000 7f567cc00700 1 -- 172.21.15.123:0/1912976629 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5670025460 con 0x7f5680107ab0 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 msgr2=0x7f568019dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 --2- 172.21.15.123:0/1912976629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5670011850 tx=0x7f5670011c10 comp rx=0 tx=0).stop 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 shutdown_connections 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 --2- 172.21.15.123:0/1912976629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5680107ab0 0x7f568019dcb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 >> 172.21.15.123:0/1912976629 conn(0x7f5680103110 msgr2=0x7f5680103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 shutdown_connections 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:47.934+0000 7f5685c00700 1 -- 172.21.15.123:0/1912976629 wait complete. 2024-03-24T00:26:48.668 INFO:teuthology.orchestra.run.smithi123.stdout:mgr not available, waiting (2/15)... 2024-03-24T00:26:48.908 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:48.791+0000 7f791f1f4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:26:48.909 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:48.908+0000 7f791f1f4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:26:49.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:49.162+0000 7f791f1f4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:26:49.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:49.403+0000 7f791f1f4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:26:50.081 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:49.771+0000 7f791f1f4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:26:50.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:50.081+0000 7f791f1f4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:26:50.818 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:50.380+0000 7f791f1f4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:26:51.790 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:51 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3447689526' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:26:51.790 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:51.423+0000 7f791f1f4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:26:52.143 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:26:52.144 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:26:52.153 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:26:52.154 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:26:34.341914+0000", 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.262+0000 7f394e800700 1 Processor -- start 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.262+0000 7f394e800700 1 -- start start 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.263+0000 7f394e800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.263+0000 7f394e800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394807c840 con 0x7f394807dd90 2024-03-24T00:26:52.155 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.263+0000 7f3947400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.263+0000 7f3947400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35500/0 (socket says 172.21.15.123:35500) 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.263+0000 7f3947400700 1 -- 172.21.15.123:0/2486670829 learned_addr learned my addr 172.21.15.123:0/2486670829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.264+0000 7f3947400700 1 -- 172.21.15.123:0/2486670829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394807c9c0 con 0x7f394807dd90 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.264+0000 7f3947400700 1 --2- 172.21.15.123:0/2486670829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39380098a0 tx=0x7f3938009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255b6fca16a35802 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.264+0000 7f3946000700 1 -- 172.21.15.123:0/2486670829 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3938004160 con 0x7f394807dd90 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f3946000700 1 -- 172.21.15.123:0/2486670829 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3938004300 con 0x7f394807dd90 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f3946000700 1 -- 172.21.15.123:0/2486670829 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3938011b50 con 0x7f394807dd90 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f394e800700 1 -- 172.21.15.123:0/2486670829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 msgr2=0x7f394807c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f394e800700 1 --2- 172.21.15.123:0/2486670829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39380098a0 tx=0x7f3938009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f394e800700 1 -- 172.21.15.123:0/2486670829 shutdown_connections 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f394e800700 1 --2- 172.21.15.123:0/2486670829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394807c300 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.265+0000 7f394e800700 1 -- 172.21.15.123:0/2486670829 >> 172.21.15.123:0/2486670829 conn(0x7f39481032c0 msgr2=0x7f39481056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.266+0000 7f394e800700 1 -- 172.21.15.123:0/2486670829 shutdown_connections 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.266+0000 7f394e800700 1 -- 172.21.15.123:0/2486670829 wait complete. 2024-03-24T00:26:52.156 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.267+0000 7f394e800700 1 Processor -- start 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.267+0000 7f394e800700 1 -- start start 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.267+0000 7f394e800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.267+0000 7f394e800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394810e110 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.268+0000 7f3947400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.268+0000 7f3947400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35512/0 (socket says 172.21.15.123:35512) 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.268+0000 7f3947400700 1 -- 172.21.15.123:0/3447689526 learned_addr learned my addr 172.21.15.123:0/3447689526 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.268+0000 7f3947400700 1 -- 172.21.15.123:0/3447689526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394819e320 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.268+0000 7f3947400700 1 --2- 172.21.15.123:0/3447689526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3938009fb0 tx=0x7f3938011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39380038b0 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3938011e50 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3938019420 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394819e5b0 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39481a12a0 con 0x7f394807dd90 2024-03-24T00:26:52.157 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3938019a60 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.269+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3938024730 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.270+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948069290 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.272+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3938043dd0 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.352+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f39481a17f0 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.353+0000 7f393fe00700 1 -- 172.21.15.123:0/3447689526 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f3938032030 con 0x7f394807dd90 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 msgr2=0x7f394819dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 --2- 172.21.15.123:0/3447689526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3938009fb0 tx=0x7f3938011bf0 comp rx=0 tx=0).stop 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 shutdown_connections 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 --2- 172.21.15.123:0/3447689526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f394807dd90 0x7f394819dde0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 >> 172.21.15.123:0/3447689526 conn(0x7f39481032c0 msgr2=0x7f3948103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 shutdown_connections 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:51.354+0000 7f394e800700 1 -- 172.21.15.123:0/3447689526 wait complete. 2024-03-24T00:26:52.158 INFO:teuthology.orchestra.run.smithi123.stdout:mgr not available, waiting (3/15)... 2024-03-24T00:26:52.523 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:52.119+0000 7f791f1f4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:26:52.523 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:52.249+0000 7f791f1f4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:26:52.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:52.523+0000 7f791f1f4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:26:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: Activating manager daemon x 2024-03-24T00:26:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: mgrmap e2: x(active, starting, since 0.0103762s) 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: Manager daemon x is now available 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' 2024-03-24T00:26:52.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:52 smithi123 ceph-mon[31487]: from='mgr.14100 172.21.15.123:0/2169450233' entity='mgr.x' 2024-03-24T00:26:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:54 smithi123 ceph-mon[31487]: mgrmap e3: x(active, since 1.02087s) 2024-03-24T00:26:55.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:55 smithi123 ceph-mon[31487]: mgrmap e4: x(active, since 2s) 2024-03-24T00:26:55.847 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1430242932' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:26:55.855 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:26:55.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:26:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:26:55.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:26:55.861 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:26:55.862 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:26:55.862 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:26:55.863 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:26:55.863 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.863 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:26:55.863 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:55.863 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:26:55.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:26:55.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:26:55.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:26:55.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:26:55.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:26:55.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:26:55.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:26:55.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:26:55.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:26:55.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:55.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:26:55.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:26:34.341914+0000", 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.802+0000 7f2c58200700 1 Processor -- start 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.802+0000 7f2c58200700 1 -- start start 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.803+0000 7f2c58200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.803+0000 7f2c58200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501085c0 con 0x7f2c50107c60 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.803+0000 7f2c55800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.803+0000 7f2c55800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35588/0 (socket says 172.21.15.123:35588) 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.803+0000 7f2c55800700 1 -- 172.21.15.123:0/57600835 learned_addr learned my addr 172.21.15.123:0/57600835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.804+0000 7f2c55800700 1 -- 172.21.15.123:0/57600835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c50108740 con 0x7f2c50107c60 2024-03-24T00:26:55.869 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.804+0000 7f2c55800700 1 --2- 172.21.15.123:0/57600835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c440098a0 tx=0x7f2c44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8444aceb61150c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.804+0000 7f2c4fe00700 1 -- 172.21.15.123:0/57600835 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c44004160 con 0x7f2c50107c60 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.804+0000 7f2c4fe00700 1 -- 172.21.15.123:0/57600835 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c44004300 con 0x7f2c50107c60 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.804+0000 7f2c4fe00700 1 -- 172.21.15.123:0/57600835 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c44011b50 con 0x7f2c50107c60 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 -- 172.21.15.123:0/57600835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 msgr2=0x7f2c50108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 --2- 172.21.15.123:0/57600835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c440098a0 tx=0x7f2c44009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 -- 172.21.15.123:0/57600835 shutdown_connections 2024-03-24T00:26:55.870 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 --2- 172.21.15.123:0/57600835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c50108080 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 -- 172.21.15.123:0/57600835 >> 172.21.15.123:0/57600835 conn(0x7f2c501032c0 msgr2=0x7f2c501056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 -- 172.21.15.123:0/57600835 shutdown_connections 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.805+0000 7f2c58200700 1 -- 172.21.15.123:0/57600835 wait complete. 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.806+0000 7f2c58200700 1 Processor -- start 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c58200700 1 -- start start 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c58200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c58200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5010c0d0 con 0x7f2c50107c60 2024-03-24T00:26:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c55800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c55800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35600/0 (socket says 172.21.15.123:35600) 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c55800700 1 -- 172.21.15.123:0/1430242932 learned_addr learned my addr 172.21.15.123:0/1430242932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.807+0000 7f2c55800700 1 -- 172.21.15.123:0/1430242932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5019e370 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c55800700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c44011b60 tx=0x7f2c44011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c44003970 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c44003b10 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5019e600 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c44024610 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.808+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c501a12f0 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.809+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f2c4401a430 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.809+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c50069290 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.810+0000 7f2c4e000700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 0x7f2c3803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.810+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2c44011e00 con 0x7f2c50107c60 2024-03-24T00:26:55.875 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.810+0000 7f2c54e00700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 0x7f2c3803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.811+0000 7f2c54e00700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 0x7f2c3803b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c40006fd0 tx=0x7f2c40006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:54.816+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c4404c9f0 con 0x7f2c50107c60 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.117+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c501a14c0 con 0x7f2c50107c60 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.117+0000 7f2c4e000700 1 -- 172.21.15.123:0/1430242932 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f2c501a14c0 con 0x7f2c50107c60 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.121+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 msgr2=0x7f2c3803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.121+0000 7f2c58200700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 0x7f2c3803b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c40006fd0 tx=0x7f2c40006e40 comp rx=0 tx=0).stop 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.121+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 msgr2=0x7f2c5019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.121+0000 7f2c58200700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c44011b60 tx=0x7f2c44011c40 comp rx=0 tx=0).stop 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 shutdown_connections 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f2c38038f20 0x7f2c3803b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 --2- 172.21.15.123:0/1430242932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50107c60 0x7f2c5019de30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 >> 172.21.15.123:0/1430242932 conn(0x7f2c501032c0 msgr2=0x7f2c50103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 shutdown_connections 2024-03-24T00:26:55.876 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:55.122+0000 7f2c58200700 1 -- 172.21.15.123:0/1430242932 wait complete. 2024-03-24T00:26:55.877 INFO:teuthology.orchestra.run.smithi123.stdout:mgr is available 2024-03-24T00:26:57.048 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:56 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3316345190' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:26:57.350 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:57.350 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:26:57.350 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout fsid = 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:26:57.350 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:26:57.351 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.468+0000 7f6b8e200700 1 Processor -- start 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.468+0000 7f6b8e200700 1 -- start start 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.469+0000 7f6b8e200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.469+0000 7f6b8e200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8807c860 con 0x7f6b8807dd60 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.469+0000 7f6b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40832/0 (socket says 172.21.15.123:40832) 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b87400700 1 -- 172.21.15.123:0/1242090661 learned_addr learned my addr 172.21.15.123:0/1242090661 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b87400700 1 -- 172.21.15.123:0/1242090661 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8807c9e0 con 0x7f6b8807dd60 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b87400700 1 --2- 172.21.15.123:0/1242090661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b780098a0 tx=0x7f6b78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6987b85c749454bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b86000700 1 -- 172.21.15.123:0/1242090661 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b78004160 con 0x7f6b8807dd60 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b86000700 1 -- 172.21.15.123:0/1242090661 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b78004300 con 0x7f6b8807dd60 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.470+0000 7f6b86000700 1 -- 172.21.15.123:0/1242090661 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b78011b50 con 0x7f6b8807dd60 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 -- 172.21.15.123:0/1242090661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 msgr2=0x7f6b8807c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 --2- 172.21.15.123:0/1242090661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b780098a0 tx=0x7f6b78009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:57.352 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 -- 172.21.15.123:0/1242090661 shutdown_connections 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 --2- 172.21.15.123:0/1242090661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b8807c320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 -- 172.21.15.123:0/1242090661 >> 172.21.15.123:0/1242090661 conn(0x7f6b881032e0 msgr2=0x7f6b88105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 -- 172.21.15.123:0/1242090661 shutdown_connections 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.471+0000 7f6b8e200700 1 -- 172.21.15.123:0/1242090661 wait complete. 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.472+0000 7f6b8e200700 1 Processor -- start 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.472+0000 7f6b8e200700 1 -- start start 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b8e200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b8e200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b88109d20 con 0x7f6b8807dd60 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40838/0 (socket says 172.21.15.123:40838) 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b87400700 1 -- 172.21.15.123:0/3316345190 learned_addr learned my addr 172.21.15.123:0/3316345190 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.473+0000 7f6b87400700 1 -- 172.21.15.123:0/3316345190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b881a26e0 con 0x7f6b8807dd60 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b87400700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b78011b60 tx=0x7f6b78011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b78003b20 con 0x7f6b8807dd60 2024-03-24T00:26:57.353 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b78003cc0 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b881a2970 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b78024840 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.474+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b881a5660 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.475+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f6b78022020 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.475+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b88069290 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.476+0000 7f6b7fe00700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 0x7f6b6c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.476+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6b78011e00 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.476+0000 7f6b86a00700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 0x7f6b6c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.477+0000 7f6b86a00700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 0x7f6b6c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.481+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b78016260 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.700+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6b880008d0 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.701+0000 7f6b7fe00700 1 -- 172.21.15.123:0/3316345190 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+425 (secure 0 0 0) 0x7f6b7804c940 con 0x7f6b8807dd60 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 msgr2=0x7f6b6c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:57.354 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 0x7f6b6c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74006e40 comp rx=0 tx=0).stop 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 msgr2=0x7f6b881a21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b78011b60 tx=0x7f6b78011c40 comp rx=0 tx=0).stop 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 shutdown_connections 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f6b6c038ed0 0x7f6b6c03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 --2- 172.21.15.123:0/3316345190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8807dd60 0x7f6b881a21a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 >> 172.21.15.123:0/3316345190 conn(0x7f6b881032e0 msgr2=0x7f6b88199030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 shutdown_connections 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:56.704+0000 7f6b8e200700 1 -- 172.21.15.123:0/3316345190 wait complete. 2024-03-24T00:26:57.355 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling cephadm module... 2024-03-24T00:26:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:58 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1908118938' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T00:26:58.607 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: ignoring --setuser ceph since I am not root 2024-03-24T00:26:58.607 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: ignoring --setgroup ceph since I am not root 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.911+0000 7fe504200700 1 Processor -- start 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.911+0000 7fe504200700 1 -- start start 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.912+0000 7fe504200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.912+0000 7fe504200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc10a800 con 0x7fe4fc109ea0 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.912+0000 7fe501800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.912+0000 7fe501800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40842/0 (socket says 172.21.15.123:40842) 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.912+0000 7fe501800700 1 -- 172.21.15.123:0/2583121303 learned_addr learned my addr 172.21.15.123:0/2583121303 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.913+0000 7fe501800700 1 -- 172.21.15.123:0/2583121303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc10af60 con 0x7fe4fc109ea0 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.913+0000 7fe501800700 1 --2- 172.21.15.123:0/2583121303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00098a0 tx=0x7fe4f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3749ecf5cfa11d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.913+0000 7fe4fbe00700 1 -- 172.21.15.123:0/2583121303 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0004160 con 0x7fe4fc109ea0 2024-03-24T00:26:58.990 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.913+0000 7fe4fbe00700 1 -- 172.21.15.123:0/2583121303 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4f0004300 con 0x7fe4fc109ea0 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.913+0000 7fe4fbe00700 1 -- 172.21.15.123:0/2583121303 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0011b50 con 0x7fe4fc109ea0 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 -- 172.21.15.123:0/2583121303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 msgr2=0x7fe4fc10a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 --2- 172.21.15.123:0/2583121303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00098a0 tx=0x7fe4f0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 -- 172.21.15.123:0/2583121303 shutdown_connections 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 --2- 172.21.15.123:0/2583121303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc10a2c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 -- 172.21.15.123:0/2583121303 >> 172.21.15.123:0/2583121303 conn(0x7fe4fc07d2b0 msgr2=0x7fe4fc07d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:58.991 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 -- 172.21.15.123:0/2583121303 shutdown_connections 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.914+0000 7fe504200700 1 -- 172.21.15.123:0/2583121303 wait complete. 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.915+0000 7fe504200700 1 Processor -- start 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.915+0000 7fe504200700 1 -- start start 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe504200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe504200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc10e3a0 con 0x7fe4fc109ea0 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe501800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe501800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40844/0 (socket says 172.21.15.123:40844) 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe501800700 1 -- 172.21.15.123:0/1908118938 learned_addr learned my addr 172.21.15.123:0/1908118938 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.916+0000 7fe501800700 1 -- 172.21.15.123:0/1908118938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc19e300 con 0x7fe4fc109ea0 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe501800700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0011b60 tx=0x7fe4f0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0003b20 con 0x7fe4fc109ea0 2024-03-24T00:26:58.995 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4f0003cc0 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc19e590 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0024840 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.917+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc1a1280 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.918+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fe4f0022020 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.918+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc0637c0 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.918+0000 7fe4fa000700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 0x7fe4e403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.919+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe4f0011e00 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.919+0000 7fe500e00700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 0x7fe4e403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.919+0000 7fe500e00700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 0x7fe4e403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8006fd0 tx=0x7fe4e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:57.923+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4f000b3e0 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.203+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe4fc1a1450 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.257+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe4fc1a1450 con 0x7fe4fc109ea0 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.260+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 msgr2=0x7fe4e403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:58.996 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.260+0000 7fe504200700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 0x7fe4e403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8006fd0 tx=0x7fe4e8006e40 comp rx=0 tx=0).stop 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.261+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 msgr2=0x7fe4fc19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.261+0000 7fe4fa000700 1 -- 172.21.15.123:0/1908118938 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe4f001a7b0 con 0x7fe4fc109ea0 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.261+0000 7fe504200700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0011b60 tx=0x7fe4f0011c40 comp rx=0 tx=0).stop 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 shutdown_connections 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7fe4e4038ed0 0x7fe4e403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 --2- 172.21.15.123:0/1908118938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc109ea0 0x7fe4fc19ddc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 >> 172.21.15.123:0/1908118938 conn(0x7fe4fc07d2b0 msgr2=0x7fe4fc1074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 shutdown_connections 2024-03-24T00:26:58.997 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:58.262+0000 7fe504200700 1 -- 172.21.15.123:0/1908118938 wait complete. 2024-03-24T00:26:59.265 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:59.034+0000 7fab0386e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:26:59.265 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:59.168+0000 7fab0386e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:26:59.545 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:59 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1908118938' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T00:26:59.545 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:26:59 smithi123 ceph-mon[31487]: mgrmap e5: x(active, since 5s) 2024-03-24T00:27:00.202 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:26:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:26:59.980+0000 7fab0386e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:27:00.202 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:00.099+0000 7fab0386e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:27:00.516 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3169661795' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:27:00.516 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:00.288+0000 7fab0386e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:27:00.516 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:00.413+0000 7fab0386e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:27:00.528 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.570+0000 7f1cd8a00700 1 Processor -- start 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.570+0000 7f1cd8a00700 1 -- start start 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.571+0000 7f1cd8a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.571+0000 7f1cd8a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41085c0 con 0x7f1cd4107c60 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.571+0000 7f1cd1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.571+0000 7f1cd1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40866/0 (socket says 172.21.15.123:40866) 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.571+0000 7f1cd1e00700 1 -- 172.21.15.123:0/411798149 learned_addr learned my addr 172.21.15.123:0/411798149 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.572+0000 7f1cd1e00700 1 -- 172.21.15.123:0/411798149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4108740 con 0x7f1cd4107c60 2024-03-24T00:27:00.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.572+0000 7f1cd1e00700 1 --2- 172.21.15.123:0/411798149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40098a0 tx=0x7f1cc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09517f4d65e0e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.572+0000 7f1cd0a00700 1 -- 172.21.15.123:0/411798149 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cc4004160 con 0x7f1cd4107c60 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.572+0000 7f1cd0a00700 1 -- 172.21.15.123:0/411798149 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cc4004300 con 0x7f1cd4107c60 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.572+0000 7f1cd0a00700 1 -- 172.21.15.123:0/411798149 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cc4011b50 con 0x7f1cd4107c60 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 -- 172.21.15.123:0/411798149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 msgr2=0x7f1cd4108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/411798149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40098a0 tx=0x7f1cc4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 -- 172.21.15.123:0/411798149 shutdown_connections 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/411798149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd4108080 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 -- 172.21.15.123:0/411798149 >> 172.21.15.123:0/411798149 conn(0x7f1cd41032c0 msgr2=0x7f1cd41056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 -- 172.21.15.123:0/411798149 shutdown_connections 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.573+0000 7f1cd8a00700 1 -- 172.21.15.123:0/411798149 wait complete. 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.574+0000 7f1cd8a00700 1 Processor -- start 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd8a00700 1 -- start start 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd8a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:00.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd8a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd410c0d0 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40876/0 (socket says 172.21.15.123:40876) 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.575+0000 7f1cd1e00700 1 -- 172.21.15.123:0/3169661795 learned_addr learned my addr 172.21.15.123:0/3169661795 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1cd1e00700 1 -- 172.21.15.123:0/3169661795 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd419e310 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1cd1e00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4011b60 tx=0x7f1cc4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cc4003970 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cc4003b10 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cc4024610 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd419e5a0 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.576+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd41a1290 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.577+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4069290 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.577+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1cc401a9f0 con 0x7f1cd4107c60 2024-03-24T00:27:00.531 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.578+0000 7f1ccaa00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 0x7f1cb803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.578+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1cc4011e00 con 0x7f1cd4107c60 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.578+0000 7f1cd1400700 1 -- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 msgr2=0x7f1cb803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.578+0000 7f1cd1400700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 0x7f1cb803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.583+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1cc4019e10 con 0x7f1cd4107c60 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.779+0000 7f1cd1400700 1 -- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 msgr2=0x7f1cb803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.779+0000 7f1cd1400700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 0x7f1cb803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.862+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1cd41a1460 con 0x7f1cd4107c60 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.862+0000 7f1ccaa00700 1 -- 172.21.15.123:0/3169661795 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1cd41a1460 con 0x7f1cd4107c60 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 msgr2=0x7f1cb803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 0x7f1cb803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 msgr2=0x7f1cd419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4011b60 tx=0x7f1cc4011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 shutdown_connections 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f1cb8038f70 0x7f1cb803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:00.532 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 --2- 172.21.15.123:0/3169661795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd4107c60 0x7f1cd419ddd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 >> 172.21.15.123:0/3169661795 conn(0x7f1cd41032c0 msgr2=0x7f1cd4103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.865+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 shutdown_connections 2024-03-24T00:27:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:26:59.866+0000 7f1cd8a00700 1 -- 172.21.15.123:0/3169661795 wait complete. 2024-03-24T00:27:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for the mgr to restart... 2024-03-24T00:27:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr epoch 5... 2024-03-24T00:27:00.805 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:00.524+0000 7fab0386e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:27:01.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:00.888+0000 7fab0386e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:27:01.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:01.031+0000 7fab0386e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:27:01.489 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:01.147+0000 7fab0386e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:27:01.489 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:01.270+0000 7fab0386e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:27:01.489 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:01.488+0000 7fab0386e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:27:02.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:02.738+0000 7fab0386e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:27:03.320 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:02.906+0000 7fab0386e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:27:03.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:03.319+0000 7fab0386e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:27:03.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:03.436+0000 7fab0386e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:27:04.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:03.690+0000 7fab0386e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:27:04.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:03.934+0000 7fab0386e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:27:04.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:04.302+0000 7fab0386e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:27:04.924 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:04.621+0000 7fab0386e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:27:05.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:04.922+0000 7fab0386e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:27:06.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:05.931+0000 7fab0386e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:27:07.026 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:06.617+0000 7fab0386e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:27:07.026 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:06.745+0000 7fab0386e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:27:07.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:07.025+0000 7fab0386e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:27:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: Active manager daemon x restarted 2024-03-24T00:27:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: Activating manager daemon x 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: mgrmap e6: x(active, starting, since 0.00537199s) 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: Manager daemon x is now available 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:07 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: Found migration_current of "None". Setting to last migration. 2024-03-24T00:27:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:08 smithi123 ceph-mon[31487]: mgrmap e7: x(active, since 1.01819s) 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.107+0000 7f42fb800700 1 Processor -- start 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.107+0000 7f42fb800700 1 -- start start 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.108+0000 7f42fb800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.108+0000 7f42fb800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f41085c0 con 0x7f42f4107c60 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.108+0000 7f42f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.108+0000 7f42f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40892/0 (socket says 172.21.15.123:40892) 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.108+0000 7f42f8e00700 1 -- 172.21.15.123:0/1087903703 learned_addr learned my addr 172.21.15.123:0/1087903703 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:08.743 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.109+0000 7f42f8e00700 1 -- 172.21.15.123:0/1087903703 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4108740 con 0x7f42f4107c60 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.109+0000 7f42f8e00700 1 --2- 172.21.15.123:0/1087903703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42e80098a0 tx=0x7f42e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f922f188e5b6be4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.109+0000 7f42f3400700 1 -- 172.21.15.123:0/1087903703 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e8004160 con 0x7f42f4107c60 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.109+0000 7f42f3400700 1 -- 172.21.15.123:0/1087903703 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42e8004300 con 0x7f42f4107c60 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.109+0000 7f42f3400700 1 -- 172.21.15.123:0/1087903703 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e8011b50 con 0x7f42f4107c60 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 -- 172.21.15.123:0/1087903703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 msgr2=0x7f42f4108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 --2- 172.21.15.123:0/1087903703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42e80098a0 tx=0x7f42e8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 -- 172.21.15.123:0/1087903703 shutdown_connections 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 --2- 172.21.15.123:0/1087903703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f4108080 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 -- 172.21.15.123:0/1087903703 >> 172.21.15.123:0/1087903703 conn(0x7f42f41032c0 msgr2=0x7f42f41056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 -- 172.21.15.123:0/1087903703 shutdown_connections 2024-03-24T00:27:08.744 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.110+0000 7f42fb800700 1 -- 172.21.15.123:0/1087903703 wait complete. 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.112+0000 7f42fb800700 1 Processor -- start 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.112+0000 7f42fb800700 1 -- start start 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.112+0000 7f42fb800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.112+0000 7f42fb800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f410c0d0 con 0x7f42f4107c60 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.112+0000 7f42f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40908/0 (socket says 172.21.15.123:40908) 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42f8e00700 1 -- 172.21.15.123:0/3245436401 learned_addr learned my addr 172.21.15.123:0/3245436401 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42f8e00700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f419e380 con 0x7f42f4107c60 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42f8e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42e8011b60 tx=0x7f42e8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e8003970 con 0x7f42f4107c60 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.113+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f419e610 con 0x7f42f4107c60 2024-03-24T00:27:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.114+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f41a1300 con 0x7f42f4107c60 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.114+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42e8003b10 con 0x7f42f4107c60 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.114+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e8024610 con 0x7f42f4107c60 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.115+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f42e80248e0 con 0x7f42f4107c60 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.116+0000 7f42f1600700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.116+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42dc03bb40 con 0x7f42dc038f70 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.116+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f42e8011e00 con 0x7f42f4107c60 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.116+0000 7f42f3e00700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.116+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.317+0000 7f42f3e00700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.317+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.718+0000 7f42f3e00700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:01.718+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:27:08.746 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:02.519+0000 7f42f3e00700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:02.519+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:04.121+0000 7f42f3e00700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/3441797972 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:04.121+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:07.034+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f42e8024bd0 con 0x7f42f4107c60 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:07.034+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 msgr2=0x7f42dc03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:07.034+0000 7f42f1600700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.041+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f42e804e540 con 0x7f42f4107c60 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.041+0000 7f42f1600700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 0x7f42dc03ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.041+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42e8023ec0 con 0x7f42dc03c860 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.043+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 0x7f42dc03ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.046+0000 7f42f3e00700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 0x7f42dc03ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f42e0003a10 tx=0x7f42e00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.048+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f42e8023ec0 con 0x7f42dc03c860 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.056+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f42d4002600 con 0x7f42dc03c860 2024-03-24T00:27:08.747 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.057+0000 7f42f1600700 1 -- 172.21.15.123:0/3245436401 <== mgr.14120 v2:172.21.15.123:6800/1263190299 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f42d4002600 con 0x7f42dc03c860 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 msgr2=0x7f42dc03ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 0x7f42dc03ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f42e0003a10 tx=0x7f42e00092b0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 msgr2=0x7f42f419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42e8011b60 tx=0x7f42e8011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 shutdown_connections 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f42dc03c860 0x7f42dc03ec50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f4107c60 0x7f42f419de40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 --2- 172.21.15.123:0/3245436401 >> [v2:172.21.15.123:6800/3441797972,v1:172.21.15.123:6801/3441797972] conn(0x7f42dc038f70 0x7f42dc03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 >> 172.21.15.123:0/3245436401 conn(0x7f42f41032c0 msgr2=0x7f42f4103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 shutdown_connections 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:08.058+0000 7f42fb800700 1 -- 172.21.15.123:0/3245436401 wait complete. 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:mgr epoch 5 is available 2024-03-24T00:27:08.748 INFO:teuthology.orchestra.run.smithi123.stdout:Setting orchestrator backend to cephadm... 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:07] ENGINE Bus STARTING 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:07] ENGINE Serving on http://172.21.15.123:8765 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:07] ENGINE Serving on https://172.21.15.123:7150 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:07] ENGINE Bus STARTED 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T00:27:09.305 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:09 smithi123 ceph-mon[31487]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T00:27:10.294 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:10 smithi123 ceph-mon[31487]: mgrmap e8: x(active, since 2s) 2024-03-24T00:27:10.294 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:10 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:10.294 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:10 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:10.302 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.304+0000 7f35a2a00700 1 Processor -- start 2024-03-24T00:27:10.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.304+0000 7f35a2a00700 1 -- start start 2024-03-24T00:27:10.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.304+0000 7f35a2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:10.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.304+0000 7f35a2a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c10a800 con 0x7f359c109ea0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.305+0000 7f359be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.305+0000 7f359be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60462/0 (socket says 172.21.15.123:60462) 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.305+0000 7f359be00700 1 -- 172.21.15.123:0/70805570 learned_addr learned my addr 172.21.15.123:0/70805570 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.305+0000 7f359be00700 1 -- 172.21.15.123:0/70805570 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c10af60 con 0x7f359c109ea0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f359be00700 1 --2- 172.21.15.123:0/70805570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e80 tx=0x7f358c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999f789502cf8331 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f359b400700 1 -- 172.21.15.123:0/70805570 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f358c017070 con 0x7f359c109ea0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f359b400700 1 -- 172.21.15.123:0/70805570 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f358c00b8c0 con 0x7f359c109ea0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f359b400700 1 -- 172.21.15.123:0/70805570 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f358c0045c0 con 0x7f359c109ea0 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f35a2a00700 1 -- 172.21.15.123:0/70805570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 msgr2=0x7f359c10a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.306+0000 7f35a2a00700 1 --2- 172.21.15.123:0/70805570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e80 tx=0x7f358c00b330 comp rx=0 tx=0).stop 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.307+0000 7f35a2a00700 1 -- 172.21.15.123:0/70805570 shutdown_connections 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.307+0000 7f35a2a00700 1 --2- 172.21.15.123:0/70805570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c10a2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.307+0000 7f35a2a00700 1 -- 172.21.15.123:0/70805570 >> 172.21.15.123:0/70805570 conn(0x7f359c07d2b0 msgr2=0x7f359c07d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:10.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.307+0000 7f35a2a00700 1 -- 172.21.15.123:0/70805570 shutdown_connections 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.307+0000 7f35a2a00700 1 -- 172.21.15.123:0/70805570 wait complete. 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.308+0000 7f35a2a00700 1 Processor -- start 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.308+0000 7f35a2a00700 1 -- start start 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.308+0000 7f35a2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.308+0000 7f35a2a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c10e3a0 con 0x7f359c109ea0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f359be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f359be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60472/0 (socket says 172.21.15.123:60472) 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f359be00700 1 -- 172.21.15.123:0/46906066 learned_addr learned my addr 172.21.15.123:0/46906066 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f359be00700 1 -- 172.21.15.123:0/46906066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c19e2e0 con 0x7f359c109ea0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f359be00700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f358c004350 tx=0x7f358c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f358c020020 con 0x7f359c109ea0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f358c003c10 con 0x7f359c109ea0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c19e570 con 0x7f359c109ea0 2024-03-24T00:27:10.310 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.309+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f358c017040 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.310+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1a1220 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.311+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0637c0 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.311+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f358c023020 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.312+0000 7f3599600700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 0x7f358003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.312+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f358c00f050 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.312+0000 7f3593400700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 0x7f358003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.313+0000 7f3593400700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 0x7f358003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3584006fd0 tx=0x7f3584006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.318+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f358c053080 con 0x7f359c109ea0 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.558+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f359c107b80 con 0x7f3580038f20 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.574+0000 7f3599600700 1 -- 172.21.15.123:0/46906066 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f359c107b80 con 0x7f3580038f20 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.577+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 msgr2=0x7f358003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.577+0000 7f35a2a00700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 0x7f358003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3584006fd0 tx=0x7f3584006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.577+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 msgr2=0x7f359c19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.577+0000 7f35a2a00700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f358c004350 tx=0x7f358c00bd50 comp rx=0 tx=0).stop 2024-03-24T00:27:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 shutdown_connections 2024-03-24T00:27:10.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3580038f20 0x7f358003b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:10.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 --2- 172.21.15.123:0/46906066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c109ea0 0x7f359c19dda0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:10.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 >> 172.21.15.123:0/46906066 conn(0x7f359c07d2b0 msgr2=0x7f359c107470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:10.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 shutdown_connections 2024-03-24T00:27:10.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:09.578+0000 7f35a2a00700 1 -- 172.21.15.123:0/46906066 wait complete. 2024-03-24T00:27:11.143 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:11 smithi123 ceph-mon[31487]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.907+0000 7f0385000700 1 Processor -- start 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.907+0000 7f0385000700 1 -- start start 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.907+0000 7f0385000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.907+0000 7f0385000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380082480 con 0x7f038010ecb0 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.908+0000 7f037de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.908+0000 7f037de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60476/0 (socket says 172.21.15.123:60476) 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.908+0000 7f037de00700 1 -- 172.21.15.123:0/2705800324 learned_addr learned my addr 172.21.15.123:0/2705800324 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:11.856 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.908+0000 7f037de00700 1 -- 172.21.15.123:0/2705800324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03801115e0 con 0x7f038010ecb0 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.908+0000 7f037de00700 1 --2- 172.21.15.123:0/2705800324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03700098a0 tx=0x7f0370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8af873194ee2307 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f037d400700 1 -- 172.21.15.123:0/2705800324 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0370004160 con 0x7f038010ecb0 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f037d400700 1 -- 172.21.15.123:0/2705800324 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0370004300 con 0x7f038010ecb0 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f037d400700 1 -- 172.21.15.123:0/2705800324 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0370011b50 con 0x7f038010ecb0 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f0385000700 1 -- 172.21.15.123:0/2705800324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 msgr2=0x7f03801110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f0385000700 1 --2- 172.21.15.123:0/2705800324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03700098a0 tx=0x7f0370009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f0385000700 1 -- 172.21.15.123:0/2705800324 shutdown_connections 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f0385000700 1 --2- 172.21.15.123:0/2705800324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801110a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.909+0000 7f0385000700 1 -- 172.21.15.123:0/2705800324 >> 172.21.15.123:0/2705800324 conn(0x7f038007d2e0 msgr2=0x7f038007f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.910+0000 7f0385000700 1 -- 172.21.15.123:0/2705800324 shutdown_connections 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.910+0000 7f0385000700 1 -- 172.21.15.123:0/2705800324 wait complete. 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f0385000700 1 Processor -- start 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f0385000700 1 -- start start 2024-03-24T00:27:11.857 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f0385000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f0385000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380112600 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f037de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f037de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60482/0 (socket says 172.21.15.123:60482) 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.911+0000 7f037de00700 1 -- 172.21.15.123:0/2791270803 learned_addr learned my addr 172.21.15.123:0/2791270803 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f037de00700 1 -- 172.21.15.123:0/2791270803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03801a2770 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f037de00700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0370011b60 tx=0x7f0370011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0370003970 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0370003b10 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0370024610 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03801a2a00 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.912+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03801a56f0 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.914+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f037001a9f0 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.914+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0380069290 con 0x7f038010ecb0 2024-03-24T00:27:11.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.914+0000 7f0376a00700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 0x7f036403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.914+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0370011e00 con 0x7f038010ecb0 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.914+0000 7f0377e00700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 0x7f036403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.915+0000 7f0377e00700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 0x7f036403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0368006fd0 tx=0x7f0368006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:10.920+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0370019de0 con 0x7f038010ecb0 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.142+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f038007e550 con 0x7f0364038f70 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.144+0000 7f0376a00700 1 -- 172.21.15.123:0/2791270803 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f038007e550 con 0x7f0364038f70 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 msgr2=0x7f036403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 0x7f036403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0368006fd0 tx=0x7f0368006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 msgr2=0x7f03801a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0370011b60 tx=0x7f0370011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 shutdown_connections 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f0364038f70 0x7f036403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 --2- 172.21.15.123:0/2791270803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f038010ecb0 0x7f03801a2230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 >> 172.21.15.123:0/2791270803 conn(0x7f038007d2e0 msgr2=0x7f038007de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:11.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.147+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 shutdown_connections 2024-03-24T00:27:11.860 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:11.148+0000 7f0385000700 1 -- 172.21.15.123:0/2791270803 wait complete. 2024-03-24T00:27:11.860 INFO:teuthology.orchestra.run.smithi123.stdout:Generating ssh key... 2024-03-24T00:27:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-mon[31487]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: Generating public/private rsa key pair. 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: Your identification has been saved in /tmp/tmpij1ur2_n/key. 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: Your public key has been saved in /tmp/tmpij1ur2_n/key.pub. 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: The key fingerprint is: 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: SHA256:TO5mK1i2mLGJX7Qb+2eCjIpXRUw0jDjgB8GvWyRjgSE ceph-082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:27:13.108 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: The key's randomart image is: 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: +---[RSA 3072]----+ 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: |E=.. *+ | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: |+o+ . +. | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: | .oo . . | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: | +.o .+ | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: |. = .. S | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: | . .o.oo | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: | oo %+o+ | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: | oo B.=*..o | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: |...o. ooo+ | 2024-03-24T00:27:13.109 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: +----[SHA256]-----+ 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.454+0000 7f0208a00700 1 Processor -- start 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.454+0000 7f0208a00700 1 -- start start 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.454+0000 7f0208a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.454+0000 7f0208a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0204082480 con 0x7f020410ecb0 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.455+0000 7f0201e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.455+0000 7f0201e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60484/0 (socket says 172.21.15.123:60484) 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.455+0000 7f0201e00700 1 -- 172.21.15.123:0/931392205 learned_addr learned my addr 172.21.15.123:0/931392205 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.455+0000 7f0201e00700 1 -- 172.21.15.123:0/931392205 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02041115e0 con 0x7f020410ecb0 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.455+0000 7f0201e00700 1 --2- 172.21.15.123:0/931392205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f01f40098a0 tx=0x7f01f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1765d4f7df6eaa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.456+0000 7f0200a00700 1 -- 172.21.15.123:0/931392205 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f4004160 con 0x7f020410ecb0 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.456+0000 7f0200a00700 1 -- 172.21.15.123:0/931392205 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01f4004300 con 0x7f020410ecb0 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.456+0000 7f0200a00700 1 -- 172.21.15.123:0/931392205 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f4011b50 con 0x7f020410ecb0 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.456+0000 7f0208a00700 1 -- 172.21.15.123:0/931392205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 msgr2=0x7f02041110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.456+0000 7f0208a00700 1 --2- 172.21.15.123:0/931392205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f01f40098a0 tx=0x7f01f4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.457+0000 7f0208a00700 1 -- 172.21.15.123:0/931392205 shutdown_connections 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.457+0000 7f0208a00700 1 --2- 172.21.15.123:0/931392205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041110a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.457+0000 7f0208a00700 1 -- 172.21.15.123:0/931392205 >> 172.21.15.123:0/931392205 conn(0x7f020407d2e0 msgr2=0x7f020407f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.457+0000 7f0208a00700 1 -- 172.21.15.123:0/931392205 shutdown_connections 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.457+0000 7f0208a00700 1 -- 172.21.15.123:0/931392205 wait complete. 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0208a00700 1 Processor -- start 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0208a00700 1 -- start start 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0208a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0208a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0204112600 con 0x7f020410ecb0 2024-03-24T00:27:13.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0201e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.458+0000 7f0201e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60500/0 (socket says 172.21.15.123:60500) 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f0201e00700 1 -- 172.21.15.123:0/3322417669 learned_addr learned my addr 172.21.15.123:0/3322417669 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f0201e00700 1 -- 172.21.15.123:0/3322417669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02041a2770 con 0x7f020410ecb0 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f0201e00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f01f4011b60 tx=0x7f01f4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f4003970 con 0x7f020410ecb0 2024-03-24T00:27:13.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01f4003b10 con 0x7f020410ecb0 2024-03-24T00:27:13.481 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f4024610 con 0x7f020410ecb0 2024-03-24T00:27:13.481 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.459+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02041a2a00 con 0x7f020410ecb0 2024-03-24T00:27:13.481 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.460+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02041a56f0 con 0x7f020410ecb0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.461+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0204069290 con 0x7f020410ecb0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.461+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f01f401a9f0 con 0x7f020410ecb0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.461+0000 7f01faa00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 0x7f01e803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.461+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f01f4011e00 con 0x7f020410ecb0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.462+0000 7f0201400700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 0x7f01e803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.462+0000 7f0201400700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 0x7f01e803b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006fd0 tx=0x7f01f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.467+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01f4019e10 con 0x7f020410ecb0 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.686+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f020407e550 con 0x7f01e8038f70 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.823+0000 7f01faa00700 1 -- 172.21.15.123:0/3322417669 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f020407e550 con 0x7f01e8038f70 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 msgr2=0x7f01e803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 0x7f01e803b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006fd0 tx=0x7f01f0006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 msgr2=0x7f02041a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f01f4011b60 tx=0x7f01f4011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:13.482 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 shutdown_connections 2024-03-24T00:27:13.483 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f01e8038f70 0x7f01e803b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:13.483 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 --2- 172.21.15.123:0/3322417669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f020410ecb0 0x7f02041a2230 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:13.483 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 >> 172.21.15.123:0/3322417669 conn(0x7f020407d2e0 msgr2=0x7f020407de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:13.483 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 shutdown_connections 2024-03-24T00:27:13.483 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:12.826+0000 7f0208a00700 1 -- 172.21.15.123:0/3322417669 wait complete. 2024-03-24T00:27:14.009 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:13 smithi123 ceph-mon[31487]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:14.009 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:13 smithi123 ceph-mon[31487]: Generating ssh key... 2024-03-24T00:27:14.009 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:13 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:14.009 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:13 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:14.920 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:14 smithi123 ceph-mon[31487]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:14.928 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:27:14.928 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.045+0000 7f759cc00700 1 Processor -- start 2024-03-24T00:27:14.928 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f759cc00700 1 -- start start 2024-03-24T00:27:14.929 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f759cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f759cc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759810a7c0 con 0x7f7598109e60 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f7595e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f7595e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60506/0 (socket says 172.21.15.123:60506) 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.046+0000 7f7595e00700 1 -- 172.21.15.123:0/3920131370 learned_addr learned my addr 172.21.15.123:0/3920131370 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.047+0000 7f7595e00700 1 -- 172.21.15.123:0/3920131370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759810af70 con 0x7f7598109e60 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.047+0000 7f7595e00700 1 --2- 172.21.15.123:0/3920131370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75880098a0 tx=0x7f7588009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28408b80aa467ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:14.930 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f7595400700 1 -- 172.21.15.123:0/3920131370 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7588004160 con 0x7f7598109e60 2024-03-24T00:27:14.932 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f7595400700 1 -- 172.21.15.123:0/3920131370 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7588004300 con 0x7f7598109e60 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f7595400700 1 -- 172.21.15.123:0/3920131370 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7588011b50 con 0x7f7598109e60 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 -- 172.21.15.123:0/3920131370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 msgr2=0x7f759810a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 --2- 172.21.15.123:0/3920131370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75880098a0 tx=0x7f7588009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 -- 172.21.15.123:0/3920131370 shutdown_connections 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 --2- 172.21.15.123:0/3920131370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759810a280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 -- 172.21.15.123:0/3920131370 >> 172.21.15.123:0/3920131370 conn(0x7f759807d340 msgr2=0x7f759807d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 -- 172.21.15.123:0/3920131370 shutdown_connections 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.048+0000 7f759cc00700 1 -- 172.21.15.123:0/3920131370 wait complete. 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.049+0000 7f759cc00700 1 Processor -- start 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f759cc00700 1 -- start start 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f759cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f759cc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759810e3b0 con 0x7f7598109e60 2024-03-24T00:27:14.933 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f7595e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f7595e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60516/0 (socket says 172.21.15.123:60516) 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f7595e00700 1 -- 172.21.15.123:0/1784267800 learned_addr learned my addr 172.21.15.123:0/1784267800 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.050+0000 7f7595e00700 1 -- 172.21.15.123:0/1784267800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759819e330 con 0x7f7598109e60 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f7595e00700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7588009fb0 tx=0x7f7588011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75880038b0 con 0x7f7598109e60 2024-03-24T00:27:14.934 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7588011e50 con 0x7f7598109e60 2024-03-24T00:27:14.936 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7588019420 con 0x7f7598109e60 2024-03-24T00:27:14.936 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759819e5c0 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.051+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75981a12b0 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.052+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f758801a430 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.052+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7598069290 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.053+0000 7f758f400700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 0x7f757c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.053+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f758804d090 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.053+0000 7f758d400700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 0x7f757c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.054+0000 7f758d400700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 0x7f757c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7580006fd0 tx=0x7f7580006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.058+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7588016260 con 0x7f7598109e60 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.279+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7598107bd0 con 0x7f757c038f20 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.281+0000 7f758f400700 1 -- 172.21.15.123:0/1784267800 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7598107bd0 con 0x7f757c038f20 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 msgr2=0x7f757c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 0x7f757c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7580006fd0 tx=0x7f7580006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 msgr2=0x7f759819ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7588009fb0 tx=0x7f7588011bf0 comp rx=0 tx=0).stop 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 shutdown_connections 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f757c038f20 0x7f757c03b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 --2- 172.21.15.123:0/1784267800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7598109e60 0x7f759819ddf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.283+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 >> 172.21.15.123:0/1784267800 conn(0x7f759807d340 msgr2=0x7f75981074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.284+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 shutdown_connections 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:14.284+0000 7f759cc00700 1 -- 172.21.15.123:0/1784267800 wait complete. 2024-03-24T00:27:14.938 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:27:14.939 INFO:teuthology.orchestra.run.smithi123.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T00:27:14.939 INFO:teuthology.orchestra.run.smithi123.stdout:Adding host smithi123... 2024-03-24T00:27:16.452 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:16 smithi123 ceph-mon[31487]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:17.771 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:17 smithi123 ceph-mon[31487]: Deploying cephadm binary to smithi123 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Added host 'smithi123' with addr '172.21.15.123' 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.550+0000 7fcf3ec00700 1 Processor -- start 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.550+0000 7fcf3ec00700 1 -- start start 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.551+0000 7fcf3ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.551+0000 7fcf3ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381085c0 con 0x7fcf38107c60 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.551+0000 7fcf37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.551+0000 7fcf37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60524/0 (socket says 172.21.15.123:60524) 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.551+0000 7fcf37e00700 1 -- 172.21.15.123:0/2474341182 learned_addr learned my addr 172.21.15.123:0/2474341182 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.552+0000 7fcf37e00700 1 -- 172.21.15.123:0/2474341182 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38108740 con 0x7fcf38107c60 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.552+0000 7fcf37e00700 1 --2- 172.21.15.123:0/2474341182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf280098a0 tx=0x7fcf28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6d1e1df9f57206 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.552+0000 7fcf36a00700 1 -- 172.21.15.123:0/2474341182 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf28004160 con 0x7fcf38107c60 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.552+0000 7fcf36a00700 1 -- 172.21.15.123:0/2474341182 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf28004300 con 0x7fcf38107c60 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.552+0000 7fcf36a00700 1 -- 172.21.15.123:0/2474341182 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf28011b50 con 0x7fcf38107c60 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 -- 172.21.15.123:0/2474341182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 msgr2=0x7fcf38108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/2474341182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf280098a0 tx=0x7fcf28009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 -- 172.21.15.123:0/2474341182 shutdown_connections 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/2474341182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf38108080 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 -- 172.21.15.123:0/2474341182 >> 172.21.15.123:0/2474341182 conn(0x7fcf381032e0 msgr2=0x7fcf38105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 -- 172.21.15.123:0/2474341182 shutdown_connections 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.553+0000 7fcf3ec00700 1 -- 172.21.15.123:0/2474341182 wait complete. 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.554+0000 7fcf3ec00700 1 Processor -- start 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.554+0000 7fcf3ec00700 1 -- start start 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.555+0000 7fcf3ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.555+0000 7fcf3ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3810c0d0 con 0x7fcf38107c60 2024-03-24T00:27:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.555+0000 7fcf37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.555+0000 7fcf37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60530/0 (socket says 172.21.15.123:60530) 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.555+0000 7fcf37e00700 1 -- 172.21.15.123:0/1527869160 learned_addr learned my addr 172.21.15.123:0/1527869160 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf37e00700 1 -- 172.21.15.123:0/1527869160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3807e090 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf37e00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcf28011b60 tx=0x7fcf28011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf28003970 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf28003b10 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf28024610 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3807c280 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.556+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3807c7c0 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.558+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fcf2801a9f0 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.558+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380637c0 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.558+0000 7fcf34c00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 0x7fcf1c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.558+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcf28011e00 con 0x7fcf38107c60 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.558+0000 7fcf37400700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 0x7fcf1c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:22.751 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.559+0000 7fcf37400700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 0x7fcf1c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf24006fd0 tx=0x7fcf24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.564+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf2800b3e0 con 0x7fcf38107c60 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:15.789+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf38104530 con 0x7fcf1c038f70 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.031+0000 7fcf34c00700 1 -- 172.21.15.123:0/1527869160 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcf38104530 con 0x7fcf1c038f70 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 msgr2=0x7fcf1c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 0x7fcf1c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf24006fd0 tx=0x7fcf24006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 msgr2=0x7fcf3807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcf28011b60 tx=0x7fcf28011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 shutdown_connections 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fcf1c038f70 0x7fcf1c03b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 --2- 172.21.15.123:0/1527869160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf38107c60 0x7fcf3807db50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 >> 172.21.15.123:0/1527869160 conn(0x7fcf381032e0 msgr2=0x7fcf38103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 shutdown_connections 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:22.037+0000 7fcf3ec00700 1 -- 172.21.15.123:0/1527869160 wait complete. 2024-03-24T00:27:22.752 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying unmanaged mon service... 2024-03-24T00:27:23.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:23 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:23.348 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:23 smithi123 ceph-mon[31487]: Added host smithi123 2024-03-24T00:27:23.348 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:23 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:24.766 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:24 smithi123 ceph-mon[31487]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:24.767 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:24 smithi123 ceph-mon[31487]: Saving service mon spec with placement count:5 2024-03-24T00:27:24.767 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:24 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.399+0000 7fa6ab800700 1 Processor -- start 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.399+0000 7fa6ab800700 1 -- start start 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.400+0000 7fa6ab800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.400+0000 7fa6ab800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a407c820 con 0x7fa6a407dd70 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.400+0000 7fa6a8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.400+0000 7fa6a8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60750/0 (socket says 172.21.15.123:60750) 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.400+0000 7fa6a8e00700 1 -- 172.21.15.123:0/3659290681 learned_addr learned my addr 172.21.15.123:0/3659290681 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.401+0000 7fa6a8e00700 1 -- 172.21.15.123:0/3659290681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a407c9a0 con 0x7fa6a407dd70 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.401+0000 7fa6a8e00700 1 --2- 172.21.15.123:0/3659290681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098a0 tx=0x7fa698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519235d365d023e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:25.456 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.402+0000 7fa6a3400700 1 -- 172.21.15.123:0/3659290681 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa698004160 con 0x7fa6a407dd70 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.402+0000 7fa6a3400700 1 -- 172.21.15.123:0/3659290681 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa698004300 con 0x7fa6a407dd70 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.402+0000 7fa6a3400700 1 -- 172.21.15.123:0/3659290681 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa698011b50 con 0x7fa6a407dd70 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.402+0000 7fa6ab800700 1 -- 172.21.15.123:0/3659290681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 msgr2=0x7fa6a407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.402+0000 7fa6ab800700 1 --2- 172.21.15.123:0/3659290681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098a0 tx=0x7fa698009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.403+0000 7fa6ab800700 1 -- 172.21.15.123:0/3659290681 shutdown_connections 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.403+0000 7fa6ab800700 1 --2- 172.21.15.123:0/3659290681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a407c2e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.403+0000 7fa6ab800700 1 -- 172.21.15.123:0/3659290681 >> 172.21.15.123:0/3659290681 conn(0x7fa6a41032e0 msgr2=0x7fa6a4105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.403+0000 7fa6ab800700 1 -- 172.21.15.123:0/3659290681 shutdown_connections 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.403+0000 7fa6ab800700 1 -- 172.21.15.123:0/3659290681 wait complete. 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.404+0000 7fa6ab800700 1 Processor -- start 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.404+0000 7fa6ab800700 1 -- start start 2024-03-24T00:27:25.457 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.404+0000 7fa6ab800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:25.458 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.404+0000 7fa6ab800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a410e110 con 0x7fa6a407dd70 2024-03-24T00:27:25.458 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.405+0000 7fa6a8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:25.458 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.405+0000 7fa6a8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60754/0 (socket says 172.21.15.123:60754) 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.405+0000 7fa6a8e00700 1 -- 172.21.15.123:0/4246530755 learned_addr learned my addr 172.21.15.123:0/4246530755 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.405+0000 7fa6a8e00700 1 -- 172.21.15.123:0/4246530755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a419e300 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.405+0000 7fa6a8e00700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa698011670 tx=0x7fa698011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.406+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6980039c0 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.406+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa698011e50 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.406+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a419e590 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.406+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa69801a5c0 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.406+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a41a1280 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.407+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa698024430 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.407+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a40637c0 con 0x7fa6a407dd70 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.407+0000 7fa6a1600700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 0x7fa68c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.407+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa69804d170 con 0x7fa6a407dd70 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.408+0000 7fa6a3e00700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 0x7fa68c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.408+0000 7fa6a3e00700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 0x7fa68c03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa690006fd0 tx=0x7fa690006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.413+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa698017ca0 con 0x7fa6a407dd70 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.655+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa6a4104500 con 0x7fa68c038f20 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.662+0000 7fa6a1600700 1 -- 172.21.15.123:0/4246530755 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa6a4104500 con 0x7fa68c038f20 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.664+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 msgr2=0x7fa68c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.664+0000 7fa6ab800700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 0x7fa68c03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa690006fd0 tx=0x7fa690006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.664+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 msgr2=0x7fa6a419ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.664+0000 7fa6ab800700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa698011670 tx=0x7fa698011b60 comp rx=0 tx=0).stop 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 shutdown_connections 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa68c038f20 0x7fa68c03b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:25.462 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 --2- 172.21.15.123:0/4246530755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a407dd70 0x7fa6a419ddc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:25.463 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 >> 172.21.15.123:0/4246530755 conn(0x7fa6a41032e0 msgr2=0x7fa6a4103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:25.463 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 shutdown_connections 2024-03-24T00:27:25.463 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:23.665+0000 7fa6ab800700 1 -- 172.21.15.123:0/4246530755 wait complete. 2024-03-24T00:27:25.463 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying unmanaged mgr service... 2024-03-24T00:27:26.287 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:26 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:26.287 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:26 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:26.974 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T00:27:26.975 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.011+0000 7f4bd1000700 1 Processor -- start 2024-03-24T00:27:26.975 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.011+0000 7f4bd1000700 1 -- start start 2024-03-24T00:27:26.975 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.011+0000 7f4bd1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:26.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.011+0000 7f4bd1000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc07c800 con 0x7f4bcc07dd50 2024-03-24T00:27:26.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.012+0000 7f4bc9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:26.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.012+0000 7f4bc9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38092/0 (socket says 172.21.15.123:38092) 2024-03-24T00:27:26.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.012+0000 7f4bc9e00700 1 -- 172.21.15.123:0/3800457363 learned_addr learned my addr 172.21.15.123:0/3800457363 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:26.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.012+0000 7f4bc9e00700 1 -- 172.21.15.123:0/3800457363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc07c980 con 0x7f4bcc07dd50 2024-03-24T00:27:26.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.013+0000 7f4bc9e00700 1 --2- 172.21.15.123:0/3800457363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0098a0 tx=0x7f4bbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccfdcead223d4ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:26.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.013+0000 7f4bc8a00700 1 -- 172.21.15.123:0/3800457363 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc004160 con 0x7f4bcc07dd50 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.013+0000 7f4bc8a00700 1 -- 172.21.15.123:0/3800457363 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4bbc004300 con 0x7f4bcc07dd50 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.013+0000 7f4bc8a00700 1 -- 172.21.15.123:0/3800457363 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc011b20 con 0x7f4bcc07dd50 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 -- 172.21.15.123:0/3800457363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 msgr2=0x7f4bcc07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 --2- 172.21.15.123:0/3800457363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0098a0 tx=0x7f4bbc009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 -- 172.21.15.123:0/3800457363 shutdown_connections 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 --2- 172.21.15.123:0/3800457363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc07c2c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:26.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 -- 172.21.15.123:0/3800457363 >> 172.21.15.123:0/3800457363 conn(0x7f4bcc1032e0 msgr2=0x7f4bcc105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 -- 172.21.15.123:0/3800457363 shutdown_connections 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.014+0000 7f4bd1000700 1 -- 172.21.15.123:0/3800457363 wait complete. 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.015+0000 7f4bd1000700 1 Processor -- start 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.015+0000 7f4bd1000700 1 -- start start 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bd1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bd1000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc10e170 con 0x7f4bcc07dd50 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bc9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bc9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38104/0 (socket says 172.21.15.123:38104) 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bc9e00700 1 -- 172.21.15.123:0/2557304027 learned_addr learned my addr 172.21.15.123:0/2557304027 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:26.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.016+0000 7f4bc9e00700 1 -- 172.21.15.123:0/2557304027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc19e290 con 0x7f4bcc07dd50 2024-03-24T00:27:26.983 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bc9e00700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0117e0 tx=0x7f4bbc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:26.983 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc003ba0 con 0x7f4bcc07dd50 2024-03-24T00:27:26.983 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4bbc01a910 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc19e520 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc0248f0 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.017+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc19e980 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.018+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc0637c0 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.019+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4bbc022020 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.019+0000 7f4bc2a00700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 0x7f4bb003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.019+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4bbc04d3c0 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.019+0000 7f4bc9400700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 0x7f4bb003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.020+0000 7f4bc9400700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 0x7f4bb003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8006fd0 tx=0x7f4bb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.025+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4bbc01ab70 con 0x7f4bcc07dd50 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.287+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4bcc1044e0 con 0x7f4bb0038f70 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.292+0000 7f4bc2a00700 1 -- 172.21.15.123:0/2557304027 <== mgr.14120 v2:172.21.15.123:6800/1263190299 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4bcc1044e0 con 0x7f4bb0038f70 2024-03-24T00:27:26.984 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 msgr2=0x7f4bb003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 0x7f4bb003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8006fd0 tx=0x7f4bb8006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 msgr2=0x7f4bcc19dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0117e0 tx=0x7f4bbc011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 shutdown_connections 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f4bb0038f70 0x7f4bb003b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 --2- 172.21.15.123:0/2557304027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bcc07dd50 0x7f4bcc19dd50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.295+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 >> 172.21.15.123:0/2557304027 conn(0x7f4bcc1032e0 msgr2=0x7f4bcc103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.296+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 shutdown_connections 2024-03-24T00:27:26.985 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:26.296+0000 7f4bd1000700 1 -- 172.21.15.123:0/2557304027 wait complete. 2024-03-24T00:27:27.574 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:27 smithi123 ceph-mon[31487]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:27.574 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:27 smithi123 ceph-mon[31487]: Saving service mgr spec with placement count:2 2024-03-24T00:27:27.575 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:27 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:28 smithi123 ceph-mon[31487]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:27:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:28 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1798003572' entity='client.admin' 2024-03-24T00:27:29.167 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.574+0000 7f5545a00700 1 Processor -- start 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.574+0000 7f5545a00700 1 -- start start 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.575+0000 7f5545a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.575+0000 7f5545a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554007be90 con 0x7f55401064c0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.575+0000 7f553ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.575+0000 7f553ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38108/0 (socket says 172.21.15.123:38108) 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.575+0000 7f553ea00700 1 -- 172.21.15.123:0/2530418756 learned_addr learned my addr 172.21.15.123:0/2530418756 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.576+0000 7f553ea00700 1 -- 172.21.15.123:0/2530418756 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5540106e20 con 0x7f55401064c0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.576+0000 7f553ea00700 1 --2- 172.21.15.123:0/2530418756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f55300098a0 tx=0x7f5530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22aa836346864a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.576+0000 7f553e000700 1 -- 172.21.15.123:0/2530418756 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5530004160 con 0x7f55401064c0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f553e000700 1 -- 172.21.15.123:0/2530418756 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5530004300 con 0x7f55401064c0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f553e000700 1 -- 172.21.15.123:0/2530418756 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5530011b20 con 0x7f55401064c0 2024-03-24T00:27:29.168 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f5545a00700 1 -- 172.21.15.123:0/2530418756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 msgr2=0x7f55401068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f5545a00700 1 --2- 172.21.15.123:0/2530418756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f55300098a0 tx=0x7f5530009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f5545a00700 1 -- 172.21.15.123:0/2530418756 shutdown_connections 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f5545a00700 1 --2- 172.21.15.123:0/2530418756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f55401068e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.577+0000 7f5545a00700 1 -- 172.21.15.123:0/2530418756 >> 172.21.15.123:0/2530418756 conn(0x7f5540102100 msgr2=0x7f5540104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.578+0000 7f5545a00700 1 -- 172.21.15.123:0/2530418756 shutdown_connections 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.578+0000 7f5545a00700 1 -- 172.21.15.123:0/2530418756 wait complete. 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f5545a00700 1 Processor -- start 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f5545a00700 1 -- start start 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f5545a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f5545a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5540107ef0 con 0x7f55401064c0 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f553ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f553ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38116/0 (socket says 172.21.15.123:38116) 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.579+0000 7f553ea00700 1 -- 172.21.15.123:0/1798003572 learned_addr learned my addr 172.21.15.123:0/1798003572 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f553ea00700 1 -- 172.21.15.123:0/1798003572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554019e340 con 0x7f55401064c0 2024-03-24T00:27:29.169 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f553ea00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5530011b60 tx=0x7f5530011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5530003b60 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f553001a910 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554019e5d0 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.580+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5530024930 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.581+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554019ea30 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.582+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5540069290 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.582+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5530022020 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.583+0000 7f5537400700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 0x7f552403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.583+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f553004d080 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.583+0000 7f5537e00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 0x7f552403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.583+0000 7f5537e00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 0x7f552403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5528006fd0 tx=0x7f5528006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.588+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f553000b3e0 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.815+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5540062da0 con 0x7f55401064c0 2024-03-24T00:27:29.170 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.820+0000 7f5537400700 1 -- 172.21.15.123:0/1798003572 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5530016260 con 0x7f55401064c0 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 msgr2=0x7f552403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 0x7f552403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5528006fd0 tx=0x7f5528006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 msgr2=0x7f554019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5530011b60 tx=0x7f5530011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 shutdown_connections 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f5524038f20 0x7f552403b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 --2- 172.21.15.123:0/1798003572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55401064c0 0x7f554019de00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 >> 172.21.15.123:0/1798003572 conn(0x7f5540102100 msgr2=0x7f5540194a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 shutdown_connections 2024-03-24T00:27:29.171 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:27.823+0000 7f5545a00700 1 -- 172.21.15.123:0/1798003572 wait complete. 2024-03-24T00:27:30.314 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:29 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:30.314 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:29 smithi123 ceph-mon[31487]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:27:30.693 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.721+0000 7f3def200700 1 Processor -- start 2024-03-24T00:27:30.693 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.721+0000 7f3def200700 1 -- start start 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.722+0000 7f3def200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.722+0000 7f3def200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8108670 con 0x7f3de8107c80 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.722+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.722+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38132/0 (socket says 172.21.15.123:38132) 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.722+0000 7f3de7e00700 1 -- 172.21.15.123:0/1876517175 learned_addr learned my addr 172.21.15.123:0/1876517175 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.723+0000 7f3de7e00700 1 -- 172.21.15.123:0/1876517175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8108e30 con 0x7f3de8107c80 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.723+0000 7f3de7e00700 1 --2- 172.21.15.123:0/1876517175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098a0 tx=0x7f3dd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b373c2861d5a2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.723+0000 7f3de6a00700 1 -- 172.21.15.123:0/1876517175 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8004160 con 0x7f3de8107c80 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.723+0000 7f3de6a00700 1 -- 172.21.15.123:0/1876517175 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dd8004300 con 0x7f3de8107c80 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3de6a00700 1 -- 172.21.15.123:0/1876517175 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8011b20 con 0x7f3de8107c80 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 -- 172.21.15.123:0/1876517175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 msgr2=0x7f3de81080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 --2- 172.21.15.123:0/1876517175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098a0 tx=0x7f3dd8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:30.694 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 -- 172.21.15.123:0/1876517175 shutdown_connections 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 --2- 172.21.15.123:0/1876517175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de81080a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 -- 172.21.15.123:0/1876517175 >> 172.21.15.123:0/1876517175 conn(0x7f3de8103220 msgr2=0x7f3de8105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 -- 172.21.15.123:0/1876517175 shutdown_connections 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.724+0000 7f3def200700 1 -- 172.21.15.123:0/1876517175 wait complete. 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3def200700 1 Processor -- start 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3def200700 1 -- start start 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3def200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3def200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de810c1e0 con 0x7f3de8107c80 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38138/0 (socket says 172.21.15.123:38138) 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.726+0000 7f3de7e00700 1 -- 172.21.15.123:0/3860143438 learned_addr learned my addr 172.21.15.123:0/3860143438 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3de7e00700 1 -- 172.21.15.123:0/3860143438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de819e3d0 con 0x7f3de8107c80 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3de7e00700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8000c00 tx=0x7f3dd8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8003b60 con 0x7f3de8107c80 2024-03-24T00:27:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dd801a910 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de819e660 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.727+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8024930 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.728+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de819eac0 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.729+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3dd8022020 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.729+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de80637c0 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.729+0000 7f3de4c00700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 0x7f3dcc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.729+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3dd804cfd0 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.730+0000 7f3de7400700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 0x7f3dcc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.730+0000 7f3de7400700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 0x7f3dcc03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006fd0 tx=0x7f3dd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.735+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3dd8016260 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.963+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f3de81a1350 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.967+0000 7f3de4c00700 1 -- 172.21.15.123:0/3860143438 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f3dd801ab70 con 0x7f3de8107c80 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.969+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 msgr2=0x7f3dcc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 0x7f3dcc03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006fd0 tx=0x7f3dd4006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:30.696 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 msgr2=0x7f3de819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8000c00 tx=0x7f3dd8011c10 comp rx=0 tx=0).stop 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 shutdown_connections 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f3dcc038f70 0x7f3dcc03b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 --2- 172.21.15.123:0/3860143438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de8107c80 0x7f3de819de90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 >> 172.21.15.123:0/3860143438 conn(0x7f3de8103220 msgr2=0x7f3de8103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 shutdown_connections 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:29.970+0000 7f3def200700 1 -- 172.21.15.123:0/3860143438 wait complete. 2024-03-24T00:27:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling the dashboard module... 2024-03-24T00:27:31.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:30 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3860143438' entity='client.admin' 2024-03-24T00:27:32.179 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-mon[31487]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:27:32.179 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-mon[31487]: from='mgr.14120 172.21.15.123:0/4023164217' entity='mgr.x' 2024-03-24T00:27:32.179 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3039162044' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T00:27:32.474 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: ignoring --setuser ceph since I am not root 2024-03-24T00:27:32.475 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: ignoring --setgroup ceph since I am not root 2024-03-24T00:27:32.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.281+0000 7fa000800700 1 Processor -- start 2024-03-24T00:27:32.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.281+0000 7fa000800700 1 -- start start 2024-03-24T00:27:32.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.282+0000 7fa000800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:32.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.282+0000 7fa000800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff807c820 con 0x7f9ff807dd70 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.282+0000 7f9ffde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.282+0000 7f9ffde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38142/0 (socket says 172.21.15.123:38142) 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.282+0000 7f9ffde00700 1 -- 172.21.15.123:0/4059749421 learned_addr learned my addr 172.21.15.123:0/4059749421 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.283+0000 7f9ffde00700 1 -- 172.21.15.123:0/4059749421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff807c9a0 con 0x7f9ff807dd70 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.283+0000 7f9ffde00700 1 --2- 172.21.15.123:0/4059749421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0098a0 tx=0x7f9fec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba96a32b05336f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.283+0000 7f9ffca00700 1 -- 172.21.15.123:0/4059749421 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fec004160 con 0x7f9ff807dd70 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.283+0000 7f9ffca00700 1 -- 172.21.15.123:0/4059749421 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fec004300 con 0x7f9ff807dd70 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.283+0000 7f9ffca00700 1 -- 172.21.15.123:0/4059749421 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fec011b20 con 0x7f9ff807dd70 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 -- 172.21.15.123:0/4059749421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 msgr2=0x7f9ff807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 --2- 172.21.15.123:0/4059749421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0098a0 tx=0x7f9fec009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 -- 172.21.15.123:0/4059749421 shutdown_connections 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 --2- 172.21.15.123:0/4059749421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff807c2e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 -- 172.21.15.123:0/4059749421 >> 172.21.15.123:0/4059749421 conn(0x7f9ff81032e0 msgr2=0x7f9ff8105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:32.817 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 -- 172.21.15.123:0/4059749421 shutdown_connections 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.284+0000 7fa000800700 1 -- 172.21.15.123:0/4059749421 wait complete. 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.285+0000 7fa000800700 1 Processor -- start 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.285+0000 7fa000800700 1 -- start start 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7fa000800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7fa000800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff810e150 con 0x7f9ff807dd70 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7f9ffde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7f9ffde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38158/0 (socket says 172.21.15.123:38158) 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7f9ffde00700 1 -- 172.21.15.123:0/3039162044 learned_addr learned my addr 172.21.15.123:0/3039162044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.286+0000 7f9ffde00700 1 -- 172.21.15.123:0/3039162044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff819e250 con 0x7f9ff807dd70 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7f9ffde00700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0117e0 tx=0x7f9fec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fec003ba0 con 0x7f9ff807dd70 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fec01a910 con 0x7f9ff807dd70 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fec0248f0 con 0x7f9ff807dd70 2024-03-24T00:27:32.818 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff819e4e0 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.287+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff819e940 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.288+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff80637c0 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.289+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9fec022020 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.290+0000 7f9ff2a00700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 0x7f9fe003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.290+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9fec04d610 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.290+0000 7f9ffd400700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 0x7f9fe003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.291+0000 7f9ffd400700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 0x7f9fe003b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8006fd0 tx=0x7f9fe8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.294+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9fec00be50 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:31.579+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f9ff81a13a0 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.063+0000 7f9ff2a00700 1 -- 172.21.15.123:0/3039162044 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f9fec01ab70 con 0x7f9ff807dd70 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 msgr2=0x7f9fe003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 0x7f9fe003b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8006fd0 tx=0x7f9fe8006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 msgr2=0x7f9ff819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:32.819 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0117e0 tx=0x7f9fec011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 shutdown_connections 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7f9fe0038f70 0x7f9fe003b430 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8006fd0 tx=0x7f9fe8006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 --2- 172.21.15.123:0/3039162044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff807dd70 0x7f9ff819dd10 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0117e0 tx=0x7f9fec011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.069+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 >> 172.21.15.123:0/3039162044 conn(0x7f9ff81032e0 msgr2=0x7f9ff8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.070+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 shutdown_connections 2024-03-24T00:27:32.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:32.070+0000 7fa000800700 1 -- 172.21.15.123:0/3039162044 wait complete. 2024-03-24T00:27:33.064 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:32.881+0000 7f0c25060200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:27:33.064 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:33.011+0000 7f0c25060200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:27:33.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:33 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3039162044' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T00:27:33.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:33 smithi123 ceph-mon[31487]: mgrmap e9: x(active, since 25s) 2024-03-24T00:27:33.936 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:33.816+0000 7f0c25060200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:27:34.230 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:33.946+0000 7f0c25060200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:27:34.230 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.130+0000 7f0c25060200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:27:34.585 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.265+0000 7f0c25060200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:27:34.586 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.376+0000 7f0c25060200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:27:34.586 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3073166980' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:27:34.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.713+0000 7f0c25060200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:27:34.941 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.936+0000 7fa881800700 1 Processor -- start 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.936+0000 7fa881800700 1 -- start start 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.937+0000 7fa881800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.937+0000 7fa881800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c07c800 con 0x7fa87c07dd50 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.937+0000 7fa87aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.937+0000 7fa87aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38188/0 (socket says 172.21.15.123:38188) 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.937+0000 7fa87aa00700 1 -- 172.21.15.123:0/4190094905 learned_addr learned my addr 172.21.15.123:0/4190094905 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.938+0000 7fa87aa00700 1 -- 172.21.15.123:0/4190094905 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c07c980 con 0x7fa87c07dd50 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.938+0000 7fa87aa00700 1 --2- 172.21.15.123:0/4190094905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0098a0 tx=0x7fa86c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd33e4a73f05a4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.938+0000 7fa879600700 1 -- 172.21.15.123:0/4190094905 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa86c004160 con 0x7fa87c07dd50 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.938+0000 7fa879600700 1 -- 172.21.15.123:0/4190094905 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa86c004300 con 0x7fa87c07dd50 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.938+0000 7fa879600700 1 -- 172.21.15.123:0/4190094905 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa86c011b20 con 0x7fa87c07dd50 2024-03-24T00:27:34.942 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 -- 172.21.15.123:0/4190094905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 msgr2=0x7fa87c07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 --2- 172.21.15.123:0/4190094905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0098a0 tx=0x7fa86c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 -- 172.21.15.123:0/4190094905 shutdown_connections 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 --2- 172.21.15.123:0/4190094905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c07c2c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 -- 172.21.15.123:0/4190094905 >> 172.21.15.123:0/4190094905 conn(0x7fa87c1032c0 msgr2=0x7fa87c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 -- 172.21.15.123:0/4190094905 shutdown_connections 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.939+0000 7fa881800700 1 -- 172.21.15.123:0/4190094905 wait complete. 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.940+0000 7fa881800700 1 Processor -- start 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.940+0000 7fa881800700 1 -- start start 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa881800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa881800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c10e190 con 0x7fa87c07dd50 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa87aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa87aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38196/0 (socket says 172.21.15.123:38196) 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa87aa00700 1 -- 172.21.15.123:0/3073166980 learned_addr learned my addr 172.21.15.123:0/3073166980 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:34.943 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa87aa00700 1 -- 172.21.15.123:0/3073166980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c19e2d0 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.941+0000 7fa87aa00700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa86c011b60 tx=0x7fa86c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.942+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa86c003b60 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.942+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa86c01a910 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.942+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa86c024930 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.942+0000 7fa881800700 1 -- 172.21.15.123:0/3073166980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c19e560 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.942+0000 7fa881800700 1 -- 172.21.15.123:0/3073166980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c1a1210 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.943+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fa86c022020 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.944+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa854005330 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.944+0000 7fa873400700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 0x7fa86003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.944+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa86c04d500 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.944+0000 7fa87a000700 1 -- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 msgr2=0x7fa86003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.944+0000 7fa87a000700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 0x7fa86003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:33.949+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa86c04ce50 con 0x7fa87c07dd50 2024-03-24T00:27:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.145+0000 7fa87a000700 1 -- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 msgr2=0x7fa86003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.145+0000 7fa87a000700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 0x7fa86003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.230+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa854005e30 con 0x7fa87c07dd50 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.230+0000 7fa873400700 1 -- 172.21.15.123:0/3073166980 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa86c01a610 con 0x7fa87c07dd50 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.234+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 msgr2=0x7fa86003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.234+0000 7fa870c00700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 0x7fa86003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.234+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 msgr2=0x7fa87c19dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.234+0000 7fa870c00700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa86c011b60 tx=0x7fa86c011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 shutdown_connections 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fa860038fd0 0x7fa86003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 --2- 172.21.15.123:0/3073166980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c07dd50 0x7fa87c19dd90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 >> 172.21.15.123:0/3073166980 conn(0x7fa87c1032c0 msgr2=0x7fa87c103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 shutdown_connections 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:34.235+0000 7fa870c00700 1 -- 172.21.15.123:0/3073166980 wait complete. 2024-03-24T00:27:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for the mgr to restart... 2024-03-24T00:27:34.946 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr epoch 9... 2024-03-24T00:27:35.110 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.856+0000 7f0c25060200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:27:35.111 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:34.977+0000 7f0c25060200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:27:35.111 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:35.110+0000 7f0c25060200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:27:35.368 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:35.331+0000 7f0c25060200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:27:36.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:36.602+0000 7f0c25060200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:27:36.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:36.772+0000 7f0c25060200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:27:37.559 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:37.187+0000 7f0c25060200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:27:37.559 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:37.305+0000 7f0c25060200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:27:37.559 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:37.558+0000 7f0c25060200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:27:37.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:37.802+0000 7f0c25060200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:27:38.516 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:38.174+0000 7f0c25060200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:27:38.517 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:38.516+0000 7f0c25060200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:27:39.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:38.816+0000 7f0c25060200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:27:40.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:39.827+0000 7f0c25060200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:27:40.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:40.515+0000 7f0c25060200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:27:40.856 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:40.642+0000 7f0c25060200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:27:40.921+0000 7f0c25060200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: Active manager daemon x restarted 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: Activating manager daemon x 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: mgrmap e10: x(active, starting, since 0.0096401s) 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:27:41.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:27:41.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:27:41.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:27:41.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:27:41.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:40 smithi123 ceph-mon[31487]: Manager daemon x is now available 2024-03-24T00:27:42.344 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:42.344 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:42.344 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:27:42.345 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:27:42.345 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:41 smithi123 ceph-mon[31487]: mgrmap e11: x(active, since 1.01138s) 2024-03-24T00:27:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:27:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T00:27:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:27:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.518+0000 7fc533200700 1 Processor -- start 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.518+0000 7fc533200700 1 -- start start 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.519+0000 7fc533200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.519+0000 7fc533200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5341084a0 con 0x7fc534107ab0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.519+0000 7fc531e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.519+0000 7fc531e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38210/0 (socket says 172.21.15.123:38210) 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.519+0000 7fc531e00700 1 -- 172.21.15.123:0/3640843876 learned_addr learned my addr 172.21.15.123:0/3640843876 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.520+0000 7fc531e00700 1 -- 172.21.15.123:0/3640843876 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534108c10 con 0x7fc534107ab0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.520+0000 7fc531e00700 1 --2- 172.21.15.123:0/3640843876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc5240098a0 tx=0x7fc524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9573d85f7b37cef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.520+0000 7fc530a00700 1 -- 172.21.15.123:0/3640843876 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc524004160 con 0x7fc534107ab0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.520+0000 7fc530a00700 1 -- 172.21.15.123:0/3640843876 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc524004300 con 0x7fc534107ab0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.520+0000 7fc530a00700 1 -- 172.21.15.123:0/3640843876 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc524011b20 con 0x7fc534107ab0 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 -- 172.21.15.123:0/3640843876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 msgr2=0x7fc534107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:42.666 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 --2- 172.21.15.123:0/3640843876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc5240098a0 tx=0x7fc524009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 -- 172.21.15.123:0/3640843876 shutdown_connections 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 --2- 172.21.15.123:0/3640843876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc534107ed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 -- 172.21.15.123:0/3640843876 >> 172.21.15.123:0/3640843876 conn(0x7fc534103050 msgr2=0x7fc534105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 -- 172.21.15.123:0/3640843876 shutdown_connections 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.521+0000 7fc533200700 1 -- 172.21.15.123:0/3640843876 wait complete. 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc533200700 1 Processor -- start 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc533200700 1 -- start start 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc533200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc533200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53410bfc0 con 0x7fc534107ab0 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc531e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc531e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38214/0 (socket says 172.21.15.123:38214) 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.523+0000 7fc531e00700 1 -- 172.21.15.123:0/1449841500 learned_addr learned my addr 172.21.15.123:0/1449841500 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc531e00700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53419bf00 con 0x7fc534107ab0 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc531e00700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc5240117e0 tx=0x7fc524011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc524003bf0 con 0x7fc534107ab0 2024-03-24T00:27:42.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52401a910 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5240249a0 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.524+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53419c190 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.525+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53419c5f0 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.526+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fc524022020 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.526+0000 7fc52aa00700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.527+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc51803bb50 con 0x7fc518038f80 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.527+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc52404d1e0 con 0x7fc534107ab0 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.527+0000 7fc531400700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.527+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.728+0000 7fc531400700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:35.728+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:36.129+0000 7fc531400700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:36.129+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:36.930+0000 7fc531400700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:42.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:36.930+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:38.532+0000 7fc531400700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1263190299 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:38.532+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:40.930+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7fc524019ca0 con 0x7fc534107ab0 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:40.931+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 msgr2=0x7fc51803b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:40.931+0000 7fc52aa00700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.931+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fc52404eeb0 con 0x7fc534107ab0 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.932+0000 7fc52aa00700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 0x7fc51803ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.932+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc524001d30 con 0x7fc51803ca30 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.933+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 0x7fc51803ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.934+0000 7fc531400700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 0x7fc51803ee20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc520003a10 tx=0x7fc520009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.936+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fc524001d30 con 0x7fc51803ca30 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.943+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc534069290 con 0x7fc51803ca30 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc52aa00700 1 -- 172.21.15.123:0/1449841500 <== mgr.14152 v2:172.21.15.123:6800/2933502234 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc534069290 con 0x7fc51803ca30 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 msgr2=0x7fc51803ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 0x7fc51803ee20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc520003a10 tx=0x7fc520009270 comp rx=0 tx=0).stop 2024-03-24T00:27:42.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 msgr2=0x7fc53419b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc5240117e0 tx=0x7fc524011c10 comp rx=0 tx=0).stop 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 shutdown_connections 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc534107ab0 0x7fc53419b9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc51803ca30 0x7fc51803ee20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 --2- 172.21.15.123:0/1449841500 >> [v2:172.21.15.123:6800/1263190299,v1:172.21.15.123:6801/1263190299] conn(0x7fc518038f80 0x7fc51803b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 >> 172.21.15.123:0/1449841500 conn(0x7fc534103050 msgr2=0x7fc534103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 shutdown_connections 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:41.944+0000 7fc533200700 1 -- 172.21.15.123:0/1449841500 wait complete. 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:mgr epoch 9 is available 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi123.stdout:Generating a dashboard self-signed certificate... 2024-03-24T00:27:43.197 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:42 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:42] ENGINE Bus STARTING 2024-03-24T00:27:44.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:43 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:42] ENGINE Serving on https://172.21.15.123:7150 2024-03-24T00:27:44.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:43 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:42] ENGINE Serving on http://172.21.15.123:8765 2024-03-24T00:27:44.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:43 smithi123 ceph-mon[31487]: [24/Mar/2024:00:27:42] ENGINE Bus STARTED 2024-03-24T00:27:44.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:43 smithi123 ceph-mon[31487]: mgrmap e12: x(active, since 2s) 2024-03-24T00:27:44.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.859+0000 7fc6ae200700 1 Processor -- start 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.859+0000 7fc6ae200700 1 -- start start 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.860+0000 7fc6ae200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.860+0000 7fc6ae200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a807be50 con 0x7fc6a8108700 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.860+0000 7fc6a7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.860+0000 7fc6a7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57004/0 (socket says 172.21.15.123:57004) 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.860+0000 7fc6a7400700 1 -- 172.21.15.123:0/3076577002 learned_addr learned my addr 172.21.15.123:0/3076577002 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.861+0000 7fc6a7400700 1 -- 172.21.15.123:0/3076577002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a810b030 con 0x7fc6a8108700 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.861+0000 7fc6a7400700 1 --2- 172.21.15.123:0/3076577002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6980098a0 tx=0x7fc698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b689e3f65385d12f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.861+0000 7fc6a6a00700 1 -- 172.21.15.123:0/3076577002 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc698004160 con 0x7fc6a8108700 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.861+0000 7fc6a6a00700 1 -- 172.21.15.123:0/3076577002 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc698004300 con 0x7fc6a8108700 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.861+0000 7fc6a6a00700 1 -- 172.21.15.123:0/3076577002 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc698011b20 con 0x7fc6a8108700 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.862+0000 7fc6ae200700 1 -- 172.21.15.123:0/3076577002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 msgr2=0x7fc6a810aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.862+0000 7fc6ae200700 1 --2- 172.21.15.123:0/3076577002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6980098a0 tx=0x7fc698009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.862+0000 7fc6ae200700 1 -- 172.21.15.123:0/3076577002 shutdown_connections 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.862+0000 7fc6ae200700 1 --2- 172.21.15.123:0/3076577002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a810aaf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.862+0000 7fc6ae200700 1 -- 172.21.15.123:0/3076577002 >> 172.21.15.123:0/3076577002 conn(0x7fc6a81020f0 msgr2=0x7fc6a8104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.863+0000 7fc6ae200700 1 -- 172.21.15.123:0/3076577002 shutdown_connections 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.863+0000 7fc6ae200700 1 -- 172.21.15.123:0/3076577002 wait complete. 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.863+0000 7fc6ae200700 1 Processor -- start 2024-03-24T00:27:44.905 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6ae200700 1 -- start start 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6ae200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6ae200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a810c090 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6a7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6a7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57006/0 (socket says 172.21.15.123:57006) 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6a7400700 1 -- 172.21.15.123:0/1212903344 learned_addr learned my addr 172.21.15.123:0/1212903344 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.864+0000 7fc6a7400700 1 -- 172.21.15.123:0/1212903344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a819c120 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6a7400700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc698011b60 tx=0x7fc698011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc698003b60 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc69801a910 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a819c3b0 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc698024930 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.865+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a819c810 con 0x7fc6a8108700 2024-03-24T00:27:44.906 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.866+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fc698022020 con 0x7fc6a8108700 2024-03-24T00:27:44.907 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.866+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8069290 con 0x7fc6a8108700 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.867+0000 7fc6a4c00700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 0x7fc68c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.867+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc69804d3d0 con 0x7fc6a8108700 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.867+0000 7fc69fe00700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 0x7fc68c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.868+0000 7fc69fe00700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 0x7fc68c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc690006fd0 tx=0x7fc690006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:43.873+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc69800bc40 con 0x7fc6a8108700 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.098+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6a8103340 con 0x7fc68c038f70 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.266+0000 7fc6a4c00700 1 -- 172.21.15.123:0/1212903344 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fc6a8103340 con 0x7fc68c038f70 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 msgr2=0x7fc68c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 0x7fc68c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc690006fd0 tx=0x7fc690006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 msgr2=0x7fc6a819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc698011b60 tx=0x7fc698011c40 comp rx=0 tx=0).stop 2024-03-24T00:27:44.909 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 shutdown_connections 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc68c038f70 0x7fc68c03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 --2- 172.21.15.123:0/1212903344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6a8108700 0x7fc6a819bbe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.269+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 >> 172.21.15.123:0/1212903344 conn(0x7fc6a81020f0 msgr2=0x7fc6a8102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.270+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 shutdown_connections 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:44.270+0000 7fc6ae200700 1 -- 172.21.15.123:0/1212903344 wait complete. 2024-03-24T00:27:44.910 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial admin user... 2024-03-24T00:27:45.190 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:44 smithi123 ceph-mon[31487]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:45.191 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:44 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:45.191 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:44 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Fts1kVOfu.I2xGvlwC8EKO7AOAIzd0arqZKq3mDNShbL9oImMXevS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711240065, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.504+0000 7fd695600700 1 Processor -- start 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.504+0000 7fd695600700 1 -- start start 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.505+0000 7fd695600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.505+0000 7fd695600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69007c800 con 0x7fd69007dd50 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.505+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.505+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57022/0 (socket says 172.21.15.123:57022) 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.505+0000 7fd68ea00700 1 -- 172.21.15.123:0/3658307784 learned_addr learned my addr 172.21.15.123:0/3658307784 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:46.642 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.506+0000 7fd68ea00700 1 -- 172.21.15.123:0/3658307784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69007c980 con 0x7fd69007dd50 2024-03-24T00:27:46.643 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.506+0000 7fd68ea00700 1 --2- 172.21.15.123:0/3658307784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6800098a0 tx=0x7fd680009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ff3cf5edfa3ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:46.643 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.506+0000 7fd68d600700 1 -- 172.21.15.123:0/3658307784 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd680004160 con 0x7fd69007dd50 2024-03-24T00:27:46.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.506+0000 7fd68d600700 1 -- 172.21.15.123:0/3658307784 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd680004300 con 0x7fd69007dd50 2024-03-24T00:27:46.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.506+0000 7fd68d600700 1 -- 172.21.15.123:0/3658307784 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd680011b20 con 0x7fd69007dd50 2024-03-24T00:27:46.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.507+0000 7fd695600700 1 -- 172.21.15.123:0/3658307784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 msgr2=0x7fd69007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.507+0000 7fd695600700 1 --2- 172.21.15.123:0/3658307784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6800098a0 tx=0x7fd680009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.507+0000 7fd695600700 1 -- 172.21.15.123:0/3658307784 shutdown_connections 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.507+0000 7fd695600700 1 --2- 172.21.15.123:0/3658307784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69007c2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.507+0000 7fd695600700 1 -- 172.21.15.123:0/3658307784 >> 172.21.15.123:0/3658307784 conn(0x7fd6901032e0 msgr2=0x7fd690105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.508+0000 7fd695600700 1 -- 172.21.15.123:0/3658307784 shutdown_connections 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.508+0000 7fd695600700 1 -- 172.21.15.123:0/3658307784 wait complete. 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd695600700 1 Processor -- start 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd695600700 1 -- start start 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd695600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd695600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69010e190 con 0x7fd69007dd50 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57036/0 (socket says 172.21.15.123:57036) 2024-03-24T00:27:46.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.509+0000 7fd68ea00700 1 -- 172.21.15.123:0/3234667607 learned_addr learned my addr 172.21.15.123:0/3234667607 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd68ea00700 1 -- 172.21.15.123:0/3234667607 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69019e270 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd68ea00700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6800117e0 tx=0x7fd680011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd680003ba0 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd68001a910 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69019e500 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.510+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6800248f0 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.511+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69019e960 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.512+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd690069290 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.512+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fd680022020 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.512+0000 7fd687400700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 0x7fd67403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.513+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd68004d8c0 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.513+0000 7fd68e000700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 0x7fd67403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.514+0000 7fd68e000700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 0x7fd67403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd67c006fd0 tx=0x7fd67c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.518+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd68001ab70 con 0x7fd69007dd50 2024-03-24T00:27:46.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.759+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd6901a1370 con 0x7fd674038f70 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.996+0000 7fd687400700 1 -- 172.21.15.123:0/3234667607 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fd6901a1370 con 0x7fd674038f70 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.998+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 msgr2=0x7fd67403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.998+0000 7fd695600700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 0x7fd67403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd67c006fd0 tx=0x7fd67c006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.998+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 msgr2=0x7fd69019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.998+0000 7fd695600700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6800117e0 tx=0x7fd680011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 shutdown_connections 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd674038f70 0x7fd67403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 --2- 172.21.15.123:0/3234667607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd69007dd50 0x7fd69019dd30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 >> 172.21.15.123:0/3234667607 conn(0x7fd6901032e0 msgr2=0x7fd690103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 shutdown_connections 2024-03-24T00:27:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:45.999+0000 7fd695600700 1 -- 172.21.15.123:0/3234667607 wait complete. 2024-03-24T00:27:46.651 INFO:teuthology.orchestra.run.smithi123.stdout:Fetching dashboard port number... 2024-03-24T00:27:46.933 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:46 smithi123 ceph-mon[31487]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:27:46.933 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:46.933 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:48.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1920850782' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T00:27:48.814 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.223+0000 7ffa51e00700 1 Processor -- start 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.223+0000 7ffa51e00700 1 -- start start 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.224+0000 7ffa51e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.224+0000 7ffa51e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c07e150 con 0x7ffa4c07dd30 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.224+0000 7ffa4aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.224+0000 7ffa4aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44990/0 (socket says 172.21.15.123:44990) 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.224+0000 7ffa4aa00700 1 -- 172.21.15.123:0/3710543781 learned_addr learned my addr 172.21.15.123:0/3710543781 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.225+0000 7ffa4aa00700 1 -- 172.21.15.123:0/3710543781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c07c7e0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.225+0000 7ffa4aa00700 1 --2- 172.21.15.123:0/3710543781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0098a0 tx=0x7ffa3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c08bb011273f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.225+0000 7ffa49600700 1 -- 172.21.15.123:0/3710543781 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa3c004160 con 0x7ffa4c07dd30 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.225+0000 7ffa49600700 1 -- 172.21.15.123:0/3710543781 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c004300 con 0x7ffa4c07dd30 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.225+0000 7ffa49600700 1 -- 172.21.15.123:0/3710543781 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa3c011b20 con 0x7ffa4c07dd30 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 -- 172.21.15.123:0/3710543781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 msgr2=0x7ffa4c07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 --2- 172.21.15.123:0/3710543781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0098a0 tx=0x7ffa3c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 -- 172.21.15.123:0/3710543781 shutdown_connections 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 --2- 172.21.15.123:0/3710543781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c07c2a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 -- 172.21.15.123:0/3710543781 >> 172.21.15.123:0/3710543781 conn(0x7ffa4c1032e0 msgr2=0x7ffa4c105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 -- 172.21.15.123:0/3710543781 shutdown_connections 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.226+0000 7ffa51e00700 1 -- 172.21.15.123:0/3710543781 wait complete. 2024-03-24T00:27:48.820 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.227+0000 7ffa51e00700 1 Processor -- start 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.227+0000 7ffa51e00700 1 -- start start 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa51e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa51e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c10e130 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa4aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa4aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45002/0 (socket says 172.21.15.123:45002) 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa4aa00700 1 -- 172.21.15.123:0/1920850782 learned_addr learned my addr 172.21.15.123:0/1920850782 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.228+0000 7ffa4aa00700 1 -- 172.21.15.123:0/1920850782 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c19e2e0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa4aa00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0117e0 tx=0x7ffa3c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa3c003ba0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c01a910 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c19e570 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1a1220 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.229+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa3c0248f0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.230+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ffa3c022020 con 0x7ffa4c07dd30 2024-03-24T00:27:48.821 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.230+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c0637c0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.231+0000 7ffa50a00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 0x7ffa3003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.231+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffa3c04e7a0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.231+0000 7ffa4a000700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 0x7ffa3003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.232+0000 7ffa4a000700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 0x7ffa3003b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.237+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa3c0172b0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.822 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.456+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ffa4c1a17f0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.823 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.457+0000 7ffa50a00700 1 -- 172.21.15.123:0/1920850782 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ffa3c01aac0 con 0x7ffa4c07dd30 2024-03-24T00:27:48.823 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 msgr2=0x7ffa3003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:48.823 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 0x7ffa3003b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:48.823 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 msgr2=0x7ffa4c19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0117e0 tx=0x7ffa3c011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 shutdown_connections 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffa30038fd0 0x7ffa3003b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.460+0000 7ffa51e00700 1 --2- 172.21.15.123:0/1920850782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c07dd30 0x7ffa4c19dda0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.461+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 >> 172.21.15.123:0/1920850782 conn(0x7ffa4c1032e0 msgr2=0x7ffa4c103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.461+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 shutdown_connections 2024-03-24T00:27:48.824 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:47.461+0000 7ffa51e00700 1 -- 172.21.15.123:0/1920850782 wait complete. 2024-03-24T00:27:48.851 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:27:48.851 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-24T00:27:48.875 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:27:48.875 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-24T00:27:48.875 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-24T00:27:48.875 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T00:27:48.884 INFO:teuthology.orchestra.run.smithi123.stdout:Ceph Dashboard is now available at: 2024-03-24T00:27:48.884 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:48.885 INFO:teuthology.orchestra.run.smithi123.stdout: URL: https://smithi123.front.sepia.ceph.com:8443/ 2024-03-24T00:27:48.885 INFO:teuthology.orchestra.run.smithi123.stdout: User: admin 2024-03-24T00:27:48.885 INFO:teuthology.orchestra.run.smithi123.stdout: Password: 3stki1r8bb 2024-03-24T00:27:48.885 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:48.886 INFO:teuthology.orchestra.run.smithi123.stdout:Saving cluster configuration to /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config directory 2024-03-24T00:27:48.888 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling autotune for osd_memory_target 2024-03-24T00:27:49.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:48 smithi123 ceph-mon[31487]: mgrmap e13: x(active, since 6s) 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.836+0000 7f120f200700 1 Processor -- start 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.836+0000 7f120f200700 1 -- start start 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.836+0000 7f120f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.836+0000 7f120f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121010a650 con 0x7f1210109c60 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.837+0000 7f120de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.837+0000 7f120de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45008/0 (socket says 172.21.15.123:45008) 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.837+0000 7f120de00700 1 -- 172.21.15.123:0/1582130232 learned_addr learned my addr 172.21.15.123:0/1582130232 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.837+0000 7f120de00700 1 -- 172.21.15.123:0/1582130232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121010ae10 con 0x7f1210109c60 2024-03-24T00:27:50.667 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.838+0000 7f120de00700 1 --2- 172.21.15.123:0/1582130232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12000098a0 tx=0x7f1200009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bcc9b519cbce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.838+0000 7f120ca00700 1 -- 172.21.15.123:0/1582130232 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1200004160 con 0x7f1210109c60 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.838+0000 7f120ca00700 1 -- 172.21.15.123:0/1582130232 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1200004300 con 0x7f1210109c60 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.838+0000 7f120ca00700 1 -- 172.21.15.123:0/1582130232 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1200011b20 con 0x7f1210109c60 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 -- 172.21.15.123:0/1582130232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 msgr2=0x7f121010a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 --2- 172.21.15.123:0/1582130232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12000098a0 tx=0x7f1200009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 -- 172.21.15.123:0/1582130232 shutdown_connections 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 --2- 172.21.15.123:0/1582130232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121010a080 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 -- 172.21.15.123:0/1582130232 >> 172.21.15.123:0/1582130232 conn(0x7f121007d030 msgr2=0x7f121007d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 -- 172.21.15.123:0/1582130232 shutdown_connections 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.839+0000 7f120f200700 1 -- 172.21.15.123:0/1582130232 wait complete. 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.840+0000 7f120f200700 1 Processor -- start 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.840+0000 7f120f200700 1 -- start start 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.840+0000 7f120f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.840+0000 7f120f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121010e1c0 con 0x7f1210109c60 2024-03-24T00:27:50.668 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.841+0000 7f120de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.841+0000 7f120de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45022/0 (socket says 172.21.15.123:45022) 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.841+0000 7f120de00700 1 -- 172.21.15.123:0/940698140 learned_addr learned my addr 172.21.15.123:0/940698140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.841+0000 7f120de00700 1 -- 172.21.15.123:0/940698140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121019e0e0 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.841+0000 7f120de00700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12000117e0 tx=0x7f1200011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.842+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1200003bf0 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.842+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f120001a910 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.842+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1200024970 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.842+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121019e370 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.842+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121019e7d0 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.843+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1210069290 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.843+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1200022020 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.844+0000 7f1206a00700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 0x7f11f403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.844+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f120004d3b0 con 0x7f1210109c60 2024-03-24T00:27:50.669 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.844+0000 7f120d400700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 0x7f11f403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.845+0000 7f120d400700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 0x7f11f403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:49.849+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1200010380 con 0x7f1210109c60 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.074+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f12101a11a0 con 0x7f1210109c60 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.075+0000 7f1206a00700 1 -- 172.21.15.123:0/940698140 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1200017ca0 con 0x7f1210109c60 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 msgr2=0x7f11f403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 0x7f11f403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 msgr2=0x7f121019dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12000117e0 tx=0x7f1200011ba0 comp rx=0 tx=0).stop 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 shutdown_connections 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11f4039070 0x7f11f403b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 --2- 172.21.15.123:0/940698140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1210109c60 0x7f121019dba0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 >> 172.21.15.123:0/940698140 conn(0x7f121007d030 msgr2=0x7f1210107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 shutdown_connections 2024-03-24T00:27:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:50.078+0000 7f120f200700 1 -- 172.21.15.123:0/940698140 wait complete. 2024-03-24T00:27:52.303 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.288+0000 7fb3e4c00700 1 Processor -- start 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.288+0000 7fb3e4c00700 1 -- start start 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.289+0000 7fb3e4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.289+0000 7fb3e4c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e010a6f0 con 0x7fb3e0109d00 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.289+0000 7fb3df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.289+0000 7fb3df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45026/0 (socket says 172.21.15.123:45026) 2024-03-24T00:27:52.304 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.289+0000 7fb3df400700 1 -- 172.21.15.123:0/73961828 learned_addr learned my addr 172.21.15.123:0/73961828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.290+0000 7fb3df400700 1 -- 172.21.15.123:0/73961828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e010ae60 con 0x7fb3e0109d00 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.290+0000 7fb3df400700 1 --2- 172.21.15.123:0/73961828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00098a0 tx=0x7fb3d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336567c3bf6b8ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.290+0000 7fb3de000700 1 -- 172.21.15.123:0/73961828 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0004160 con 0x7fb3e0109d00 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.290+0000 7fb3de000700 1 -- 172.21.15.123:0/73961828 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d0004300 con 0x7fb3e0109d00 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3de000700 1 -- 172.21.15.123:0/73961828 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0011b20 con 0x7fb3e0109d00 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 -- 172.21.15.123:0/73961828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 msgr2=0x7fb3e010a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/73961828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00098a0 tx=0x7fb3d0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 -- 172.21.15.123:0/73961828 shutdown_connections 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/73961828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e010a120 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 -- 172.21.15.123:0/73961828 >> 172.21.15.123:0/73961828 conn(0x7fb3e007d1e0 msgr2=0x7fb3e007d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 -- 172.21.15.123:0/73961828 shutdown_connections 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.291+0000 7fb3e4c00700 1 -- 172.21.15.123:0/73961828 wait complete. 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.292+0000 7fb3e4c00700 1 Processor -- start 2024-03-24T00:27:52.308 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3e4c00700 1 -- start start 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3e4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3e4c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e010e210 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45042/0 (socket says 172.21.15.123:45042) 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.293+0000 7fb3df400700 1 -- 172.21.15.123:0/639089256 learned_addr learned my addr 172.21.15.123:0/639089256 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3df400700 1 -- 172.21.15.123:0/639089256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e019e100 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3df400700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0000c00 tx=0x7fb3d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0003b60 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d001a910 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e019e390 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0024930 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.294+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e019e7f0 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.295+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb3d0022020 con 0x7fb3e0109d00 2024-03-24T00:27:52.309 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.295+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e0069290 con 0x7fb3e0109d00 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.296+0000 7fb3d7e00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 0x7fb3c403b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.296+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3d004d360 con 0x7fb3e0109d00 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.296+0000 7fb3dea00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 0x7fb3c403b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.297+0000 7fb3dea00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 0x7fb3c403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006fd0 tx=0x7fb3cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.302+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3d002d3f0 con 0x7fb3e0109d00 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.630+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb3e01a11c0 con 0x7fb3e0109d00 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.634+0000 7fb3d7e00700 1 -- 172.21.15.123:0/639089256 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb3d001a430 con 0x7fb3e0109d00 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 msgr2=0x7fb3c403b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 0x7fb3c403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006fd0 tx=0x7fb3cc006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:52.312 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 msgr2=0x7fb3e019dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0000c00 tx=0x7fb3d0011c10 comp rx=0 tx=0).stop 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 shutdown_connections 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3c4039010 0x7fb3c403b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 --2- 172.21.15.123:0/639089256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3e0109d00 0x7fb3e019dbc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 >> 172.21.15.123:0/639089256 conn(0x7fb3e007d1e0 msgr2=0x7fb3e0107470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 shutdown_connections 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-24T00:27:51.637+0000 7fb3e4c00700 1 -- 172.21.15.123:0/639089256 wait complete. 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.313 INFO:teuthology.orchestra.run.smithi123.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: ceph telemetry on 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout:For more information see: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:27:52.314 INFO:teuthology.orchestra.run.smithi123.stdout:Bootstrap complete. 2024-03-24T00:27:52.444 INFO:tasks.cephadm:Fetching config... 2024-03-24T00:27:52.444 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:27:52.444 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T00:27:52.492 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T00:27:52.492 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:27:52.492 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T00:27:52.572 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T00:27:52.572 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:27:52.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T00:27:52.707 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T00:27:52.707 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:27:52.707 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T00:27:52.758 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T00:27:52.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:27:52.866 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:27:52.909 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:52 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/639089256' entity='client.admin' 2024-03-24T00:27:52.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:27:52.975 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:27:52.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:27:53.037 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm6W51y3jvgt5rJvYgNwqlW3WDCgSCR4a6w3RkIJei2BXCK5GyHBgs7HSTtd0vCJEbd2YOZjmV56QrvB2IgfTFfYDPstR/hRer5v9zwfLGD+ZyhIekEjmXfdCgSDAzpr7F+IyRFoovgD1uXiXFZbXLP2snI8zFGsIxCLzJLQEQwS2mETl7hangLDhTyHFnScgFqYbnKoI+YpvJck/PjVyUEOjGTeA8Lvi7VlhSN7+RHHkSO71QC8soUp4jLstcOBHrvzEgvb/D4IW1mzLQ9RCBaddvtCC5TSd0Ua+v5dgkYEgvi1hkN6O4QOdwrfr7iXdnJOirfrIYFOOsdPuaX+Oj1us98ybnjBczSXtNhgxhhYYQlDujUyJz5hYLo5f4E19J11p63tD1q3oa35kGat+fCqy9bmnT7Bk2EiWJXBqcVL+2PBNqYCXc4ahmb2RWjeclpku/7jsRTfDo9jdUMG+r08ELearc1uWo91gfArUXnER4Eyt/fVyQ878Pg2RxpoU= ceph-082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:27:53.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T00:27:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:27:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 -- 172.21.15.123:0/3339769114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 msgr2=0x7f622c076f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 --2- 172.21.15.123:0/3339769114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c076f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f621c009910 tx=0x7f621c009c20 comp rx=0 tx=0).stop 2024-03-24T00:27:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 -- 172.21.15.123:0/3339769114 shutdown_connections 2024-03-24T00:27:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 --2- 172.21.15.123:0/3339769114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c076f60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 -- 172.21.15.123:0/3339769114 >> 172.21.15.123:0/3339769114 conn(0x7f622c0ff360 msgr2=0x7f622c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.244+0000 7f6233000700 1 -- 172.21.15.123:0/3339769114 shutdown_connections 2024-03-24T00:27:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.245+0000 7f6233000700 1 -- 172.21.15.123:0/3339769114 wait complete. 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.245+0000 7f6233000700 1 Processor -- start 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f6233000700 1 -- start start 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f6233000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f6233000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c10e2b0 con 0x7f622c076b40 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f622be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f622be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42606/0 (socket says 172.21.15.123:42606) 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.246+0000 7f622be00700 1 -- 172.21.15.123:0/4144171020 learned_addr learned my addr 172.21.15.123:0/4144171020 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:27:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f622be00700 1 -- 172.21.15.123:0/4144171020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c19ddd0 con 0x7f622c076b40 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f622be00700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f621c004fa0 tx=0x7f621c005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f621c027070 con 0x7f622c076b40 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f621c00be40 con 0x7f622c076b40 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f621c00f460 con 0x7f622c076b40 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c19e060 con 0x7f622c076b40 2024-03-24T00:27:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.247+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c1a0d50 con 0x7f622c076b40 2024-03-24T00:27:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.249+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f621c02a070 con 0x7f622c076b40 2024-03-24T00:27:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.249+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c10e9c0 con 0x7f622c076b40 2024-03-24T00:27:56.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.249+0000 7f6228c00700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 0x7f621003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:27:56.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.249+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f621c01e030 con 0x7f622c076b40 2024-03-24T00:27:56.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.249+0000 7f622b400700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 0x7f621003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:27:56.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.250+0000 7f622b400700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 0x7f621003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6218006fd0 tx=0x7f6218006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:27:56.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.255+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f621c013070 con 0x7f622c076b40 2024-03-24T00:27:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.477+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f622c02db40 con 0x7f622c076b40 2024-03-24T00:27:56.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.481+0000 7f6228c00700 1 -- 172.21.15.123:0/4144171020 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f621c028260 con 0x7f622c076b40 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 msgr2=0x7f621003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 0x7f621003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6218006fd0 tx=0x7f6218006e40 comp rx=0 tx=0).stop 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 msgr2=0x7f622c19d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f621c004fa0 tx=0x7f621c005080 comp rx=0 tx=0).stop 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 shutdown_connections 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6210039020 0x7f621003b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 --2- 172.21.15.123:0/4144171020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f622c076b40 0x7f622c19d890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:27:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 >> 172.21.15.123:0/4144171020 conn(0x7f622c0ff360 msgr2=0x7f622c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:27:56.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 shutdown_connections 2024-03-24T00:27:56.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:27:56.484+0000 7f6233000700 1 -- 172.21.15.123:0/4144171020 wait complete. 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4144171020' entity='client.admin' 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:27:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:27:57.298 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T00:27:57.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T00:27:58.379 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 -- 172.21.15.123:0/1736264387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 msgr2=0x7fa19c101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/1736264387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c101fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa190009910 tx=0x7fa190009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 -- 172.21.15.123:0/1736264387 shutdown_connections 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/1736264387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c101fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 -- 172.21.15.123:0/1736264387 >> 172.21.15.123:0/1736264387 conn(0x7fa19c0fd180 msgr2=0x7fa19c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 -- 172.21.15.123:0/1736264387 shutdown_connections 2024-03-24T00:28:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.269+0000 7fa1a3a00700 1 -- 172.21.15.123:0/1736264387 wait complete. 2024-03-24T00:28:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.270+0000 7fa1a3a00700 1 Processor -- start 2024-03-24T00:28:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.270+0000 7fa1a3a00700 1 -- start start 2024-03-24T00:28:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a3a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a3a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c107c50 con 0x7fa19c101bc0 2024-03-24T00:28:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42622/0 (socket says 172.21.15.123:42622) 2024-03-24T00:28:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a1000700 1 -- 172.21.15.123:0/618649024 learned_addr learned my addr 172.21.15.123:0/618649024 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a1000700 1 -- 172.21.15.123:0/618649024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c076390 con 0x7fa19c101bc0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.271+0000 7fa1a1000700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa190005850 tx=0x7fa190005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.272+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa190026070 con 0x7fa19c101bc0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.272+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1900044a0 con 0x7fa19c101bc0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.272+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa190003ac0 con 0x7fa19c101bc0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.272+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c074580 con 0x7fa19c101bc0 2024-03-24T00:28:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.272+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c074ac0 con 0x7fa19c101bc0 2024-03-24T00:28:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.273+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa190029070 con 0x7fa19c101bc0 2024-03-24T00:28:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.273+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19c1083d0 con 0x7fa19c101bc0 2024-03-24T00:28:00.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.274+0000 7fa199600700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 0x7fa18403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:00.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.274+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa19001e070 con 0x7fa19c101bc0 2024-03-24T00:28:00.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.274+0000 7fa19be00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 0x7fa18403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:00.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.275+0000 7fa19be00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 0x7fa18403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa188006fd0 tx=0x7fa188006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:00.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.279+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa190013070 con 0x7fa19c101bc0 2024-03-24T00:28:00.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.506+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fa19c1906d0 con 0x7fa184039020 2024-03-24T00:28:00.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.513+0000 7fa199600700 1 -- 172.21.15.123:0/618649024 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa19c1906d0 con 0x7fa184039020 2024-03-24T00:28:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.516+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 msgr2=0x7fa18403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.516+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 0x7fa18403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa188006fd0 tx=0x7fa188006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.516+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 msgr2=0x7fa19c075e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:00.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.516+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa190005850 tx=0x7fa190005030 comp rx=0 tx=0).stop 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 shutdown_connections 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa184039020 0x7fa18403b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 --2- 172.21.15.123:0/618649024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19c101bc0 0x7fa19c075e50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 >> 172.21.15.123:0/618649024 conn(0x7fa19c0fd180 msgr2=0x7fa19c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 shutdown_connections 2024-03-24T00:28:00.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:00.517+0000 7fa1a3a00700 1 -- 172.21.15.123:0/618649024 wait complete. 2024-03-24T00:28:01.331 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-03-24T00:28:01.331 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:28:01.332 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:28:01.351 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:28:01.351 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:01.410 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-03-24T00:28:01.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch host add smithi177 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:01 smithi123 ceph-mon[31487]: Updating smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:01.818 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:02 smithi123 ceph-mon[31487]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:03.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 -- 172.21.15.123:0/4243511227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 msgr2=0x7fa5281087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:03.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 --2- 172.21.15.123:0/4243511227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281087d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009910 tx=0x7fa51c009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:03.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 -- 172.21.15.123:0/4243511227 shutdown_connections 2024-03-24T00:28:03.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 --2- 172.21.15.123:0/4243511227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281087d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:03.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 -- 172.21.15.123:0/4243511227 >> 172.21.15.123:0/4243511227 conn(0x7fa5280779c0 msgr2=0x7fa528079de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:03.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 -- 172.21.15.123:0/4243511227 shutdown_connections 2024-03-24T00:28:03.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.897+0000 7fa52f600700 1 -- 172.21.15.123:0/4243511227 wait complete. 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.898+0000 7fa52f600700 1 Processor -- start 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52f600700 1 -- start start 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5281127b0 con 0x7fa5281083f0 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42638/0 (socket says 172.21.15.123:42638) 2024-03-24T00:28:03.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.899+0000 7fa52cc00700 1 -- 172.21.15.123:0/2588245155 learned_addr learned my addr 172.21.15.123:0/2588245155 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa52cc00700 1 -- 172.21.15.123:0/2588245155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5281a3e70 con 0x7fa5281083f0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa52cc00700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006010 tx=0x7fa51c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa51c026070 con 0x7fa5281083f0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa51c00bbd0 con 0x7fa5281083f0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5281a4160 con 0x7fa5281083f0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa51c003910 con 0x7fa5281083f0 2024-03-24T00:28:03.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.900+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52819daf0 con 0x7fa5281083f0 2024-03-24T00:28:03.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.901+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa51c029070 con 0x7fa5281083f0 2024-03-24T00:28:03.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.902+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5281098c0 con 0x7fa5281083f0 2024-03-24T00:28:03.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.902+0000 7fa525600700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 0x7fa51003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:03.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.902+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa51c01e070 con 0x7fa5281083f0 2024-03-24T00:28:03.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.902+0000 7fa527e00700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 0x7fa51003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:03.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.903+0000 7fa527e00700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 0x7fa51003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:03.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:03.908+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa51c013070 con 0x7fa5281083f0 2024-03-24T00:28:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:03 smithi123 ceph-mon[31487]: Updating smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:28:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:03 smithi123 ceph-mon[31487]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:04.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:04.134+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) v1 -- 0x7fa52819efe0 con 0x7fa510038fc0 2024-03-24T00:28:05.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:04 smithi123 ceph-mon[31487]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:05 smithi123 ceph-mon[31487]: Deploying cephadm binary to smithi177 2024-03-24T00:28:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:05 smithi123 ceph-mon[31487]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:07 smithi123 ceph-mon[31487]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:08.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.823+0000 7fa525600700 1 -- 172.21.15.123:0/2588245155 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa52819efe0 con 0x7fa510038fc0 2024-03-24T00:28:08.824 INFO:teuthology.orchestra.run.smithi123.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 msgr2=0x7fa51003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 0x7fa51003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 msgr2=0x7fa5281a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006010 tx=0x7fa51c005030 comp rx=0 tx=0).stop 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 shutdown_connections 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa510038fc0 0x7fa51003b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 --2- 172.21.15.123:0/2588245155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5281083f0 0x7fa5281a3930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:08.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 >> 172.21.15.123:0/2588245155 conn(0x7fa5280779c0 msgr2=0x7fa528109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:08.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 shutdown_connections 2024-03-24T00:28:08.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:08.827+0000 7fa52f600700 1 -- 172.21.15.123:0/2588245155 wait complete. 2024-03-24T00:28:09.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:28:10.016 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:10.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:10.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:09 smithi123 ceph-mon[31487]: Added host smithi177 2024-03-24T00:28:10.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:10.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:09 smithi123 ceph-mon[31487]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:10.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:11.213 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.983+0000 7f12bba00700 1 -- 172.21.15.123:0/3369978086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 msgr2=0x7f12b4106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.983+0000 7f12bba00700 1 --2- 172.21.15.123:0/3369978086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b4106e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009910 tx=0x7f12a8009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.984+0000 7f12bba00700 1 -- 172.21.15.123:0/3369978086 shutdown_connections 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.984+0000 7f12bba00700 1 --2- 172.21.15.123:0/3369978086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b4106e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.984+0000 7f12bba00700 1 -- 172.21.15.123:0/3369978086 >> 172.21.15.123:0/3369978086 conn(0x7f12b40fc580 msgr2=0x7f12b40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:11.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.984+0000 7f12bba00700 1 -- 172.21.15.123:0/3369978086 shutdown_connections 2024-03-24T00:28:11.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.984+0000 7f12bba00700 1 -- 172.21.15.123:0/3369978086 wait complete. 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.985+0000 7f12bba00700 1 Processor -- start 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.985+0000 7f12bba00700 1 -- start start 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.985+0000 7f12bba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.985+0000 7f12bba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b410b090 con 0x7f12b4106a50 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.985+0000 7f12b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.986+0000 7f12b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37338/0 (socket says 172.21.15.123:37338) 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.986+0000 7f12b9000700 1 -- 172.21.15.123:0/419651436 learned_addr learned my addr 172.21.15.123:0/419651436 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.986+0000 7f12b9000700 1 -- 172.21.15.123:0/419651436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b419fb50 con 0x7f12b4106a50 2024-03-24T00:28:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.986+0000 7f12b9000700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005850 tx=0x7f12a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:11.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.986+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a8026070 con 0x7f12b4106a50 2024-03-24T00:28:11.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.987+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12a800bbd0 con 0x7f12b4106a50 2024-03-24T00:28:11.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.987+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b419fe40 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.987+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a8003970 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.987+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b4199770 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.988+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f12a8029070 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.988+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4102190 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.988+0000 7f12b1600700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 0x7f129c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.988+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f12a801e070 con 0x7f12b4106a50 2024-03-24T00:28:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.989+0000 7f12b3e00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 0x7f129c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:11.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.989+0000 7f12b3e00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 0x7f129c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:11.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:11.994+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12a8013070 con 0x7f12b4106a50 2024-03-24T00:28:12.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:11 smithi123 ceph-mon[31487]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:12.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:12.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.225+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12b419ac80 con 0x7f129c038fc0 2024-03-24T00:28:12.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.226+0000 7f12b1600700 1 -- 172.21.15.123:0/419651436 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f12b419ac80 con 0x7f129c038fc0 2024-03-24T00:28:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:28:12.226 INFO:teuthology.orchestra.run.smithi123.stdout:[{"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-03-24T00:28:12.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 msgr2=0x7f129c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:12.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 0x7f129c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:12.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 msgr2=0x7f12b419f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005850 tx=0x7f12a8005030 comp rx=0 tx=0).stop 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 shutdown_connections 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f129c038fc0 0x7f129c03b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 --2- 172.21.15.123:0/419651436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b4106a50 0x7f12b419f610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.229+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 >> 172.21.15.123:0/419651436 conn(0x7f12b40fc580 msgr2=0x7f12b4108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.230+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 shutdown_connections 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:12.230+0000 7f12bba00700 1 -- 172.21.15.123:0/419651436 wait complete. 2024-03-24T00:28:12.952 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-03-24T00:28:12.952 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:28:12.953 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:28:12.971 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:28:12.971 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:13.029 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-03-24T00:28:13.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch host add smithi204 2024-03-24T00:28:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:14.331 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:13 smithi123 ceph-mon[31487]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:28:14.331 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:13 smithi123 ceph-mon[31487]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 -- 172.21.15.123:0/1935031201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 msgr2=0x7fb984100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 --2- 172.21.15.123:0/1935031201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb984100ee0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb974009910 tx=0x7fb974009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 -- 172.21.15.123:0/1935031201 shutdown_connections 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 --2- 172.21.15.123:0/1935031201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb984100ee0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 -- 172.21.15.123:0/1935031201 >> 172.21.15.123:0/1935031201 conn(0x7fb9840fc580 msgr2=0x7fb9840fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 -- 172.21.15.123:0/1935031201 shutdown_connections 2024-03-24T00:28:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.482+0000 7fb989000700 1 -- 172.21.15.123:0/1935031201 wait complete. 2024-03-24T00:28:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.483+0000 7fb989000700 1 Processor -- start 2024-03-24T00:28:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb989000700 1 -- start start 2024-03-24T00:28:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb989000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb989000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb984112800 con 0x7fb984100b00 2024-03-24T00:28:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb981e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb981e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37360/0 (socket says 172.21.15.123:37360) 2024-03-24T00:28:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.484+0000 7fb981e00700 1 -- 172.21.15.123:0/3874877867 learned_addr learned my addr 172.21.15.123:0/3874877867 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb981e00700 1 -- 172.21.15.123:0/3874877867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98419e2b0 con 0x7fb984100b00 2024-03-24T00:28:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb981e00700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb974006010 tx=0x7fb974005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb974026070 con 0x7fb984100b00 2024-03-24T00:28:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb97400bc30 con 0x7fb984100b00 2024-03-24T00:28:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98419e540 con 0x7fb984100b00 2024-03-24T00:28:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.485+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9740039a0 con 0x7fb984100b00 2024-03-24T00:28:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.486+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98419e9a0 con 0x7fb984100b00 2024-03-24T00:28:15.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.487+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb974029070 con 0x7fb984100b00 2024-03-24T00:28:15.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.487+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98406a700 con 0x7fb984100b00 2024-03-24T00:28:15.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.487+0000 7fb97aa00700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 0x7fb96803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:15.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.487+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb97401e070 con 0x7fb984100b00 2024-03-24T00:28:15.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.488+0000 7fb981400700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 0x7fb96803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.488+0000 7fb981400700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 0x7fb96803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:15.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.493+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb974013070 con 0x7fb984100b00 2024-03-24T00:28:15.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:15.771+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}) v1 -- 0x7fb98419f0d0 con 0x7fb968038fc0 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:15.772 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:15 smithi123 ceph-mon[31487]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:17.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:17.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:28:17.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:17.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:17.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:17.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:18.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:17 smithi123 ceph-mon[31487]: Deploying cephadm binary to smithi204 2024-03-24T00:28:18.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:17 smithi123 ceph-mon[31487]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:20.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:19 smithi123 ceph-mon[31487]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:20.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.426+0000 7fb97aa00700 1 -- 172.21.15.123:0/3874877867 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb98419f0d0 con 0x7fb968038fc0 2024-03-24T00:28:20.426 INFO:teuthology.orchestra.run.smithi123.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 msgr2=0x7fb96803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 0x7fb96803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 msgr2=0x7fb98419dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb974006010 tx=0x7fb974005030 comp rx=0 tx=0).stop 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 shutdown_connections 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb968038fc0 0x7fb96803b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 --2- 172.21.15.123:0/3874877867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb984100b00 0x7fb98419dd70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 >> 172.21.15.123:0/3874877867 conn(0x7fb9840fc580 msgr2=0x7fb9840fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 shutdown_connections 2024-03-24T00:28:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:20.430+0000 7fb989000700 1 -- 172.21.15.123:0/3874877867 wait complete. 2024-03-24T00:28:21.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:21.782 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:21.782 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:21 smithi123 ceph-mon[31487]: Added host smithi204 2024-03-24T00:28:21.782 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:21.782 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:21 smithi123 ceph-mon[31487]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:21.783 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:22.791 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:23.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 -- 172.21.15.123:0/424608604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 msgr2=0x7f103c104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:23.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 --2- 172.21.15.123:0/424608604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c104390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1030009910 tx=0x7f1030009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:23.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 -- 172.21.15.123:0/424608604 shutdown_connections 2024-03-24T00:28:23.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 --2- 172.21.15.123:0/424608604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c104390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:23.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 -- 172.21.15.123:0/424608604 >> 172.21.15.123:0/424608604 conn(0x7f103c0ff880 msgr2=0x7f103c101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:23.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 -- 172.21.15.123:0/424608604 shutdown_connections 2024-03-24T00:28:23.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.551+0000 7f1043e00700 1 -- 172.21.15.123:0/424608604 wait complete. 2024-03-24T00:28:23.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.552+0000 7f1043e00700 1 Processor -- start 2024-03-24T00:28:23.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1043e00700 1 -- start start 2024-03-24T00:28:23.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1043e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:23.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1043e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c10e380 con 0x7f103c103fb0 2024-03-24T00:28:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1041400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1041400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53552/0 (socket says 172.21.15.123:53552) 2024-03-24T00:28:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.553+0000 7f1041400700 1 -- 172.21.15.123:0/497690879 learned_addr learned my addr 172.21.15.123:0/497690879 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.554+0000 7f1041400700 1 -- 172.21.15.123:0/497690879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c19fbc0 con 0x7f103c103fb0 2024-03-24T00:28:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.554+0000 7f1041400700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1030006010 tx=0x7f1030005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.554+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1030026070 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.554+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f103000bbd0 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.554+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1030003910 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.555+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c19feb0 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.555+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c199920 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.556+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c105490 con 0x7f103c103fb0 2024-03-24T00:28:23.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.556+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1030029070 con 0x7f103c103fb0 2024-03-24T00:28:23.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.556+0000 7f103a000700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 0x7f102403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:23.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.557+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f103001e070 con 0x7f103c103fb0 2024-03-24T00:28:23.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.557+0000 7f1040a00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 0x7f102403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.557+0000 7f1040a00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 0x7f102403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1028006fd0 tx=0x7f1028006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.562+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1030013070 con 0x7f103c103fb0 2024-03-24T00:28:23.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.788+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f103c19ac80 con 0x7f1024039020 2024-03-24T00:28:23.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.790+0000 7f103a000700 1 -- 172.21.15.123:0/497690879 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f103c19ac80 con 0x7f1024039020 2024-03-24T00:28:23.790 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:28:23.791 INFO:teuthology.orchestra.run.smithi123.stdout:[{"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-03-24T00:28:23.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 msgr2=0x7f102403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:23.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 0x7f102403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1028006fd0 tx=0x7f1028006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:23.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 msgr2=0x7f103c19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:23.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1030006010 tx=0x7f1030005030 comp rx=0 tx=0).stop 2024-03-24T00:28:23.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 shutdown_connections 2024-03-24T00:28:23.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1024039020 0x7f102403b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:23.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 --2- 172.21.15.123:0/497690879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f103c103fb0 0x7f103c19f680 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:23.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 >> 172.21.15.123:0/497690879 conn(0x7f103c0ff880 msgr2=0x7f103c106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:23.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 shutdown_connections 2024-03-24T00:28:23.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:23.794+0000 7f1043e00700 1 -- 172.21.15.123:0/497690879 wait complete. 2024-03-24T00:28:24.452 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:24 smithi123 ceph-mon[31487]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:24.452 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:24 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:24.507 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T00:28:24.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T00:28:24.957 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:25.389 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:25 smithi123 ceph-mon[31487]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:28:26.224 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:26 smithi123 ceph-mon[31487]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.001+0000 7fc3e0e00700 1 -- 172.21.15.123:0/1611205845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 msgr2=0x7fc3dc1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.001+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/1611205845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1087d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00b0a0 tx=0x7fc3cc00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.002+0000 7fc3e0e00700 1 -- 172.21.15.123:0/1611205845 shutdown_connections 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.002+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/1611205845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1087d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.002+0000 7fc3e0e00700 1 -- 172.21.15.123:0/1611205845 >> 172.21.15.123:0/1611205845 conn(0x7fc3dc0779c0 msgr2=0x7fc3dc079de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.002+0000 7fc3e0e00700 1 -- 172.21.15.123:0/1611205845 shutdown_connections 2024-03-24T00:28:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.002+0000 7fc3e0e00700 1 -- 172.21.15.123:0/1611205845 wait complete. 2024-03-24T00:28:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.003+0000 7fc3e0e00700 1 Processor -- start 2024-03-24T00:28:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.003+0000 7fc3e0e00700 1 -- start start 2024-03-24T00:28:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.003+0000 7fc3e0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.003+0000 7fc3e0e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc1127b0 con 0x7fc3dc1083f0 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.004+0000 7fc3d9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.004+0000 7fc3d9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35124/0 (socket says 172.21.15.123:35124) 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.004+0000 7fc3d9e00700 1 -- 172.21.15.123:0/739826401 learned_addr learned my addr 172.21.15.123:0/739826401 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.004+0000 7fc3d9e00700 1 -- 172.21.15.123:0/739826401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc1a3dd0 con 0x7fc3dc1083f0 2024-03-24T00:28:27.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.004+0000 7fc3d9e00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0093e0 tx=0x7fc3cc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.005+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3cc017040 con 0x7fc3dc1083f0 2024-03-24T00:28:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.005+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3cc022070 con 0x7fc3dc1083f0 2024-03-24T00:28:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.005+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc1a4060 con 0x7fc3dc1083f0 2024-03-24T00:28:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.005+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc19da50 con 0x7fc3dc1083f0 2024-03-24T00:28:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.005+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3cc01d440 con 0x7fc3dc1083f0 2024-03-24T00:28:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.006+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc3cc004000 con 0x7fc3dc1083f0 2024-03-24T00:28:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.006+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc1098c0 con 0x7fc3dc1083f0 2024-03-24T00:28:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.007+0000 7fc3d2a00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 0x7fc3c003b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.007+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc3cc00f050 con 0x7fc3dc1083f0 2024-03-24T00:28:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.007+0000 7fc3d9400700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 0x7fc3c003b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.008+0000 7fc3d9400700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 0x7fc3c003b4d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006fd0 tx=0x7fc3c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.012+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3cc007e20 con 0x7fc3dc1083f0 2024-03-24T00:28:27.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.288+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc3dc19ef80 con 0x7fc3dc1083f0 2024-03-24T00:28:27.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.392+0000 7fc3d2a00700 1 -- 172.21.15.123:0/739826401 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc3cc01ce20 con 0x7fc3dc1083f0 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.396+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 msgr2=0x7fc3c003b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.396+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 0x7fc3c003b4d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006fd0 tx=0x7fc3c8006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 msgr2=0x7fc3dc1a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0093e0 tx=0x7fc3cc003ce0 comp rx=0 tx=0).stop 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 shutdown_connections 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3c0039010 0x7fc3c003b4d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 --2- 172.21.15.123:0/739826401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc1083f0 0x7fc3dc1a3890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 >> 172.21.15.123:0/739826401 conn(0x7fc3dc0779c0 msgr2=0x7fc3dc109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 shutdown_connections 2024-03-24T00:28:27.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:27.397+0000 7fc3e0e00700 1 -- 172.21.15.123:0/739826401 wait complete. 2024-03-24T00:28:27.399 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2024-03-24T00:28:27.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:27.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:27.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:27.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:27.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:27.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:27 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/739826401' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:28:28.209 INFO:tasks.cephadm:Adding mon.a on smithi123 2024-03-24T00:28:28.209 INFO:tasks.cephadm:Adding mon.b on smithi177 2024-03-24T00:28:28.209 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-03-24T00:28:28.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c' 2024-03-24T00:28:28.438 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:28:28.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/739826401' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:28:28.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:28.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:28.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:28.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:29.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:29 smithi123 ceph-mon[31487]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 -- 172.21.15.204:0/3208676364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 msgr2=0x7f193c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 --2- 172.21.15.204:0/3208676364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c106e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1930009910 tx=0x7f1930009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 -- 172.21.15.204:0/3208676364 shutdown_connections 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 --2- 172.21.15.204:0/3208676364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c106e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 -- 172.21.15.204:0/3208676364 >> 172.21.15.204:0/3208676364 conn(0x7f193c0fc5a0 msgr2=0x7f193c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 -- 172.21.15.204:0/3208676364 shutdown_connections 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.912+0000 7f1940e0c700 1 -- 172.21.15.204:0/3208676364 wait complete. 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.913+0000 7f1940e0c700 1 Processor -- start 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.913+0000 7f1940e0c700 1 -- start start 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.913+0000 7f1940e0c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:29.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.913+0000 7f1940e0c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c10b0b0 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.913+0000 7f193a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.914+0000 7f193a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.204:55298/0 (socket says 172.21.15.204:55298) 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.914+0000 7f193a59c700 1 -- 172.21.15.204:0/1086042369 learned_addr learned my addr 172.21.15.204:0/1086042369 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.914+0000 7f193a59c700 1 -- 172.21.15.204:0/1086042369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c198a20 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.914+0000 7f193a59c700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1930006010 tx=0x7f1930005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.915+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1930026070 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.915+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c198d10 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.915+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c1991d0 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.915+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f193000bbd0 con 0x7f193c106a70 2024-03-24T00:28:29.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.915+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1930030060 con 0x7f193c106a70 2024-03-24T00:28:29.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.916+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c005330 con 0x7f193c106a70 2024-03-24T00:28:29.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.916+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1930029070 con 0x7f193c106a70 2024-03-24T00:28:29.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.916+0000 7f192b7fe700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 0x7f192403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:29.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.916+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f193001e070 con 0x7f193c106a70 2024-03-24T00:28:29.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.917+0000 7f1939d9b700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 0x7f192403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:29.918 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.918+0000 7f1939d9b700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 0x7f192403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:29.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:29.930+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1930013070 con 0x7f193c106a70 2024-03-24T00:28:30.138 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.139+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f191c000c80 con 0x7f1924038fc0 2024-03-24T00:28:30.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.153+0000 7f192b7fe700 1 -- 172.21.15.204:0/1086042369 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f191c000c80 con 0x7f1924038fc0 2024-03-24T00:28:30.154 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 msgr2=0x7f192403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 0x7f192403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 msgr2=0x7f193c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1930006010 tx=0x7f1930005030 comp rx=0 tx=0).stop 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 shutdown_connections 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1924038fc0 0x7f192403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 --2- 172.21.15.204:0/1086042369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f193c106a70 0x7f193c1984e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 >> 172.21.15.204:0/1086042369 conn(0x7f193c0fc5a0 msgr2=0x7f193c0fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 shutdown_connections 2024-03-24T00:28:30.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:30.156+0000 7f1940e0c700 1 -- 172.21.15.204:0/1086042369 wait complete. 2024-03-24T00:28:30.771 DEBUG:teuthology.orchestra.run.smithi177:mon.b> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T00:28:30.774 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T00:28:30.777 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:28:30.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: Saving service mon spec with placement smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c;count:3 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:32 smithi123 ceph-mon[31487]: Deploying daemon mon.c on smithi204 2024-03-24T00:28:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:32 smithi123 ceph-mon[31487]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:32.796 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:28:34.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:34 smithi123 ceph-mon[31487]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:34.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.626+0000 7fe5687ba700 1 -- 172.21.15.204:0/1831907003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 msgr2=0x7fe560105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.626+0000 7fe5687ba700 1 --2- 172.21.15.204:0/1831907003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560105f50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009910 tx=0x7fe55c009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 -- 172.21.15.204:0/1831907003 shutdown_connections 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 --2- 172.21.15.204:0/1831907003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560105f50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 -- 172.21.15.204:0/1831907003 >> 172.21.15.204:0/1831907003 conn(0x7fe5600fc2a0 msgr2=0x7fe5600fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 -- 172.21.15.204:0/1831907003 shutdown_connections 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 -- 172.21.15.204:0/1831907003 wait complete. 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.627+0000 7fe5687ba700 1 Processor -- start 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.628+0000 7fe5687ba700 1 -- start start 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.628+0000 7fe5687ba700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.628+0000 7fe5687ba700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56010b1c0 con 0x7fe560105b70 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.629+0000 7fe566556700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:34.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.629+0000 7fe566556700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.204:40822/0 (socket says 172.21.15.204:40822) 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.629+0000 7fe566556700 1 -- 172.21.15.204:0/3062995286 learned_addr learned my addr 172.21.15.204:0/3062995286 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.629+0000 7fe566556700 1 -- 172.21.15.204:0/3062995286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560196bc0 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe566556700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe55c005850 tx=0x7fe55c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c026070 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe560196e50 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56019af00 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe55c00bc30 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.630+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c030060 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.631+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe560102480 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.636+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fe55c029070 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.636+0000 7fe5577fe700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 0x7fe54c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.636+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe55c01e070 con 0x7fe560105b70 2024-03-24T00:28:34.757 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.637+0000 7fe565d55700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 0x7fe54c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:34.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.638+0000 7fe565d55700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 0x7fe54c03b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:34.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.651+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe55c013070 con 0x7fe560105b70 2024-03-24T00:28:34.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.664+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe55c00ec70 con 0x7fe560105b70 2024-03-24T00:28:34.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:34.935+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe560197990 con 0x7fe560105b70 2024-03-24T00:28:39.699 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.699+0000 7fe5577fe700 1 -- 172.21.15.204:0/3062995286 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7fe55c00bde0 con 0x7fe560105b70 2024-03-24T00:28:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:28:39.700 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","modified":"2024-03-24T00:28:34.663340Z","created":"2024-03-24T00:26:30.024868Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.702+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 msgr2=0x7fe54c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.702+0000 7fe5687ba700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 0x7fe54c03b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550006e40 comp rx=0 tx=0).stop 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.702+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 msgr2=0x7fe560196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.702+0000 7fe5687ba700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe55c005850 tx=0x7fe55c005030 comp rx=0 tx=0).stop 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 shutdown_connections 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe54c039020 0x7fe54c03b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 --2- 172.21.15.204:0/3062995286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe560105b70 0x7fe560196680 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 >> 172.21.15.204:0/3062995286 conn(0x7fe5600fc2a0 msgr2=0x7fe5601027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 shutdown_connections 2024-03-24T00:28:39.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:39.703+0000 7fe5687ba700 1 -- 172.21.15.204:0/3062995286 wait complete. 2024-03-24T00:28:39.704 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: mon.a calling monitor election 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/3062995286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: mon.c calling monitor election 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: monmap e2: 2 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: fsmap 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: mgrmap e13: x(active, since 58s) 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: overall HEALTH_OK 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:40.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:41.032 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:40 smithi123 ceph-mon[31487]: Deploying daemon mon.b on smithi177 2024-03-24T00:28:41.033 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:41.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:41.343 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:28:41.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:28:41.585 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:28:41.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:41 smithi204 ceph-mon[36345]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:42.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:41 smithi123 ceph-mon[31487]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:42.106 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:28:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:28:41.661+0000 7f0bda200700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T00:28:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 -- 172.21.15.204:0/89029966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 msgr2=0x7f61541032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 --2- 172.21.15.204:0/89029966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f61541032d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6148009910 tx=0x7f6148009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 -- 172.21.15.204:0/89029966 shutdown_connections 2024-03-24T00:28:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 --2- 172.21.15.204:0/89029966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f61541032d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:43.101 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 -- 172.21.15.204:0/89029966 >> 172.21.15.204:0/89029966 conn(0x7f61540fc6a0 msgr2=0x7f61540feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:43.102 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 -- 172.21.15.204:0/89029966 shutdown_connections 2024-03-24T00:28:43.102 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.101+0000 7f6159f60700 1 -- 172.21.15.204:0/89029966 wait complete. 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 Processor -- start 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 -- start start 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f615419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615410c340 con 0x7f6154102ef0 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6159f60700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615410c1c0 con 0x7f615419dba0 2024-03-24T00:28:43.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:41676/0 (socket says 172.21.15.204:41676) 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 -- 172.21.15.204:0/2982505899 learned_addr learned my addr 172.21.15.204:0/2982505899 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 -- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 msgr2=0x7f61541977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 -- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 msgr2=0x7f61541977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.102+0000 7f6152ffd700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.103+0000 7f61537fe700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f615419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.103+0000 7f61537fe700 1 -- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 msgr2=0x7f61541977e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.103+0000 7f61537fe700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.103+0000 7f61537fe700 1 -- 172.21.15.204:0/2982505899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615419e1e0 con 0x7f6154102ef0 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f61537fe700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f615419d660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6148014000 tx=0x7f6148005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f6148005dc0 con 0x7f6154102ef0 2024-03-24T00:28:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6154197d80 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f614800be40 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f614800f460 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.104+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61541982e0 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.105+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615406abb0 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.105+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f614802a070 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.106+0000 7f6150ff9700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 0x7f613c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.106+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f614801e030 con 0x7f6154102ef0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.106+0000 7f6152ffd700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 0x7f613c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:43.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.107+0000 7f6152ffd700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 0x7f613c03b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6144009180 tx=0x7f6144008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:43.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.121+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6148013070 con 0x7f6154102ef0 2024-03-24T00:28:43.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.394+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6154198590 con 0x7f6154102ef0 2024-03-24T00:28:43.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.395+0000 7f6150ff9700 1 -- 172.21.15.204:0/2982505899 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f6148028260 con 0x7f6154102ef0 2024-03-24T00:28:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:28:43.395 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","modified":"2024-03-24T00:28:34.663340Z","created":"2024-03-24T00:26:30.024868Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 msgr2=0x7f613c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 0x7f613c03b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6144009180 tx=0x7f6144008040 comp rx=0 tx=0).stop 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 msgr2=0x7f615419d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f615419d660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6148014000 tx=0x7f6148005030 comp rx=0 tx=0).stop 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 shutdown_connections 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f613c038d60 0x7f613c03b220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6154102ef0 0x7f615419d660 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 --2- 172.21.15.204:0/2982505899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615419dba0 0x7f61541977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 >> 172.21.15.204:0/2982505899 conn(0x7f61540fc6a0 msgr2=0x7f6154105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 shutdown_connections 2024-03-24T00:28:43.397 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:43.397+0000 7f6159f60700 1 -- 172.21.15.204:0/2982505899 wait complete. 2024-03-24T00:28:43.398 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-03-24T00:28:43.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 podman[36013]: 2024-03-24 00:28:43.387432873 +0000 UTC m=+0.126399381 container create 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 podman[36013]: 2024-03-24 00:28:43.581174566 +0000 UTC m=+0.320141092 container init 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 podman[36013]: 2024-03-24 00:28:43.584310937 +0000 UTC m=+0.323277445 container start 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pidfile_write: ignore empty --pid-file 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: load: jerasure load: lrc 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Git sha 0 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: DB SUMMARY 2024-03-24T00:28:43.935 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: DB Session ID: BFIU2MUYG0TPPBP2CZSH 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.env: 0x560ec1b22b20 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:28:43.936 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.info_log: 0x560ec3f3f920 2024-03-24T00:28:43.937 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:28:43.938 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.statistics: (nil) 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.use_fsync: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.db_log_dir: 2024-03-24T00:28:43.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.wal_dir: 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.write_buffer_manager: 0x560ec3f121e0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:28:43.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.unordered_write: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.row_cache: None 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.wal_filter: None 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:28:43.941 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.wal_compression: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:28:43.942 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_open_files: -1 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:28:43.943 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Compression algorithms supported: 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kZSTD supported: 0 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:28:43.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.merge_operator: 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_filter: None 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560ec3f3fa20) 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_type: 0 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: data_block_index_type: 0 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_shortening: 1 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:28:43.945 INFO:journalctl@ceph.mon.b.smithi177.stdout: checksum: 4 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: no_block_cache: 0 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache: 0x560ec3f45090 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_options: 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: capacity : 536870912 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: num_shard_bits : 4 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: strict_capacity_limit : 0 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_compressed: (nil) 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: persistent_cache: (nil) 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_size: 4096 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_size_deviation: 10 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_restart_interval: 16 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_block_restart_interval: 1 2024-03-24T00:28:43.946 INFO:journalctl@ceph.mon.b.smithi177.stdout: metadata_block_size: 4096 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: partition_filters: 0 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: use_delta_encoding: 1 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: filter_policy: bloomfilter 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: whole_key_filtering: 1 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: verify_compression: 0 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: format_version: 5 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: enable_index_compression: 1 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_align: 0 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: max_auto_readahead_size: 262144 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: prepopulate_block_cache: 0 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:28:43.947 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression: NoCompression 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.num_levels: 7 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:28:43.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:28:43.949 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:28:43.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:28:43.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.ttl: 2592000 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:28:43.952 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 21e33734-f963-4503-ad91-79ea2c68d144 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240123613756, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240123614196, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711240123, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "21e33734-f963-4503-ad91-79ea2c68d144", "db_session_id": "BFIU2MUYG0TPPBP2CZSH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711240123614278, "job": 1, "event": "recovery_finished"} 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T00:28:43.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560ec4038000 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: DB pointer 0x560ec4022000 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** DB Stats ** 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.954 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:28:43.955 INFO:journalctl@ceph.mon.b.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: Block cache BinnedLRUCache@0x560ec3f45090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T00:28:43.956 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: using public_addr v2:172.21.15.177:0/0 -> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: starting mon.b rank -1 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(???) e0 preinit fsid 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: e1 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: legacy client fscid: -1 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout: No filesystems configured 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T00:28:43.957 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Deploying cephadm binary to smithi177 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Added host smithi177 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:43.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Deploying cephadm binary to smithi204 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Added host smithi204 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/739826401' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:28:43.961 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/739826401' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Saving service mon spec with placement smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c;count:3 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:43.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Deploying daemon mon.c on smithi204 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.a calling monitor election 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/3062995286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.c calling monitor election 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: monmap e2: 2 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: fsmap 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mgrmap e13: x(active, since 58s) 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: overall HEALTH_OK 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: Deploying daemon mon.b on smithi177 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:43.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:43.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 ceph-mon[36034]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T00:28:43.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 bash[36013]: 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 2024-03-24T00:28:43.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:43 smithi177 systemd[1]: Started Ceph mon.b for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:28:44.982 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:28:44.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:28:45.227 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:28:49.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: mon.c calling monitor election 2024-03-24T00:28:49.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:28:49.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: mon.a calling monitor election 2024-03-24T00:28:49.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: fsmap 2024-03-24T00:28:49.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: mgrmap e13: x(active, since 67s) 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: mon.c calling monitor election 2024-03-24T00:28:49.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: mon.a calling monitor election 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: fsmap 2024-03-24T00:28:49.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: mgrmap e13: x(active, since 67s) 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.532 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- 172.21.15.204:0/4153032003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 msgr2=0x7f0d14005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:49.532 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/4153032003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d14005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d18005fa0 tx=0x7f0d18010fd0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- 172.21.15.204:0/4153032003 shutdown_connections 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/4153032003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d14005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- 172.21.15.204:0/4153032003 >> 172.21.15.204:0/4153032003 conn(0x7f0d24077280 msgr2=0x7f0d24077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- 172.21.15.204:0/4153032003 shutdown_connections 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- 172.21.15.204:0/4153032003 wait complete. 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 Processor -- start 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.530+0000 7f0d2abb3700 1 -- start start 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d241956f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24107ac0 con 0x7f0d241005a0 2024-03-24T00:28:49.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24107940 con 0x7f0d24195c30 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2abb3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24107c40 con 0x7f0d2419a500 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:47220/0 (socket says 172.21.15.204:47220) 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 -- 172.21.15.204:0/813937906 learned_addr learned my addr 172.21.15.204:0/813937906 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d23fff700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 msgr2=0x7f0d2419a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 msgr2=0x7f0d2419a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d29150700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d23fff700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 msgr2=0x7f0d24199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d23fff700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 msgr2=0x7f0d24199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d23fff700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d23fff700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2894f700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d241956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2894f700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 msgr2=0x7f0d2419a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2894f700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2894f700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 msgr2=0x7f0d24199fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.531+0000 7f0d2894f700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d2894f700 1 -- 172.21.15.204:0/813937906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d241a2270 con 0x7f0d241005a0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d2894f700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d241956f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d180056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d180129b0 con 0x7f0d241005a0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d241a2440 con 0x7f0d241005a0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241a29d0 con 0x7f0d241005a0 2024-03-24T00:28:49.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.532+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d18029070 con 0x7f0d241005a0 2024-03-24T00:28:49.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.533+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1801b410 con 0x7f0d241005a0 2024-03-24T00:28:49.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.534+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140017e0 con 0x7f0d241005a0 2024-03-24T00:28:49.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.534+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f0d18023070 con 0x7f0d241005a0 2024-03-24T00:28:49.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.534+0000 7f0d21ffb700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 0x7f0d0803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:49.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.534+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0d1804eae0 con 0x7f0d241005a0 2024-03-24T00:28:49.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.535+0000 7f0d23fff700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 0x7f0d0803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:49.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.535+0000 7f0d23fff700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 0x7f0d0803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009850 tx=0x7f0d10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:49.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.548+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d1800e3c0 con 0x7f0d241005a0 2024-03-24T00:28:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:49 smithi204 ceph-mon[36345]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:49.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.859+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d14001ad0 con 0x7f0d241005a0 2024-03-24T00:28:49.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.860+0000 7f0d21ffb700 1 -- 172.21.15.204:0/813937906 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7f0d1802de20 con 0x7f0d241005a0 2024-03-24T00:28:49.860 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-24T00:28:49.861 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","modified":"2024-03-24T00:28:43.636420Z","created":"2024-03-24T00:26:30.024868Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 msgr2=0x7f0d0803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 0x7f0d0803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009850 tx=0x7f0d10008040 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 msgr2=0x7f0d241956f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d241956f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d180056c0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 shutdown_connections 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0d08039450 0x7f0d0803b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d2419a500 0x7f0d2419a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d241005a0 0x7f0d241956f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.862+0000 7f0d2abb3700 1 --2- 172.21.15.204:0/813937906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d24195c30 0x7f0d24199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.863+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 >> 172.21.15.204:0/813937906 conn(0x7f0d24077280 msgr2=0x7f0d241033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.863+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 shutdown_connections 2024-03-24T00:28:49.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:49.863+0000 7f0d2abb3700 1 -- 172.21.15.204:0/813937906 wait complete. 2024-03-24T00:28:49.864 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-03-24T00:28:50.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:49 smithi123 ceph-mon[31487]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:50.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:50.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: mon.c calling monitor election 2024-03-24T00:28:50.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:28:50.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: mon.a calling monitor election 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: fsmap 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: mgrmap e13: x(active, since 67s) 2024-03-24T00:28:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:50.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:51.064 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T00:28:51.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T00:28:51.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mon.b calling monitor election 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mon.b calling monitor election 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mon.c calling monitor election 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mon.a calling monitor election 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: fsmap 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: mgrmap e13: x(active, since 69s) 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Cluster is now healthy 2024-03-24T00:28:51.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: overall HEALTH_OK 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:51.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:51 smithi204 ceph-mon[36345]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:51.478 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mon.b calling monitor election 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mon.b calling monitor election 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mon.c calling monitor election 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mon.a calling monitor election 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:28:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: fsmap 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: mgrmap e13: x(active, since 69s) 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Cluster is now healthy 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: overall HEALTH_OK 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-24T00:28:51.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-24T00:28:51.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:51.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:51.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:51 smithi123 ceph-mon[31487]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mon.b calling monitor election 2024-03-24T00:28:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mon.b calling monitor election 2024-03-24T00:28:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mon.c calling monitor election 2024-03-24T00:28:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mon.a calling monitor election 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: monmap e3: 3 mons at {a=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: fsmap 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: mgrmap e13: x(active, since 69s) 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Cluster is now healthy 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: overall HEALTH_OK 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:51.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Updating smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:51.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:51 smithi177 ceph-mon[36034]: Updating smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:52.639 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.639 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: Updating smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:52.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: Updating smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:52.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:52.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: Updating smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:28:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:52.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:53.701 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:53 smithi123 ceph-mon[31487]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:28:53.701 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:53 smithi123 ceph-mon[31487]: Reconfiguring daemon mon.a on smithi123 2024-03-24T00:28:53.701 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:53 smithi123 ceph-mon[31487]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:53 smithi177 ceph-mon[36034]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:28:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:53 smithi177 ceph-mon[36034]: Reconfiguring daemon mon.a on smithi123 2024-03-24T00:28:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:53 smithi177 ceph-mon[36034]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:53.723 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:53 smithi204 ceph-mon[36345]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:28:53.723 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:53 smithi204 ceph-mon[36345]: Reconfiguring daemon mon.a on smithi123 2024-03-24T00:28:53.723 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:53 smithi204 ceph-mon[36345]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:54.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.860+0000 7f6538400700 1 -- 172.21.15.123:0/2977584614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65300745c0 msgr2=0x7f65300749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:54.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.860+0000 7f6538400700 1 --2- 172.21.15.123:0/2977584614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65300745c0 0x7f65300749a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6524009910 tx=0x7f6524009c20 comp rx=0 tx=0).stop 2024-03-24T00:28:54.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.861+0000 7f6538400700 1 -- 172.21.15.123:0/2977584614 shutdown_connections 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.861+0000 7f6538400700 1 --2- 172.21.15.123:0/2977584614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65300745c0 0x7f65300749a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.861+0000 7f6538400700 1 -- 172.21.15.123:0/2977584614 >> 172.21.15.123:0/2977584614 conn(0x7f65300fd660 msgr2=0x7f65300ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.861+0000 7f6538400700 1 -- 172.21.15.123:0/2977584614 shutdown_connections 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.861+0000 7f6538400700 1 -- 172.21.15.123:0/2977584614 wait complete. 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.862+0000 7f6538400700 1 Processor -- start 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.862+0000 7f6538400700 1 -- start start 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.862+0000 7f6538400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 0x7f653019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.862+0000 7f6538400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 0x7f65301999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.862+0000 7f6538400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 0x7f653019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6538400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653010e400 con 0x7f653019fd20 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6538400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653010e280 con 0x7f6530199ee0 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6538400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653010e580 con 0x7f65300745c0 2024-03-24T00:28:54.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 0x7f65301999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 0x7f65301999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37384/0 (socket says 172.21.15.123:37384) 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535000700 1 -- 172.21.15.123:0/3295868760 learned_addr learned my addr 172.21.15.123:0/3295868760 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 0x7f653019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6536400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 0x7f653019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 -- 172.21.15.123:0/3295868760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 msgr2=0x7f653019a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 0x7f653019a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 -- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 msgr2=0x7f65301999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 0x7f65301999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6535a00700 1 -- 172.21.15.123:0/3295868760 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65301a66b0 con 0x7f65300745c0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.863+0000 7f6536400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 0x7f653019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f6535a00700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 0x7f653019f7e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6524005950 tx=0x7f6524004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524026070 con 0x7f65300745c0 2024-03-24T00:28:54.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f652400b750 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65301a6880 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524003a00 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.864+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65301a6db0 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.865+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6510005330 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.865+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f6524029070 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.866+0000 7f652e000700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 0x7f651803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.866+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f652404cb10 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.866+0000 7f6535000700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 0x7f651803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.867+0000 7f6535000700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 0x7f651803b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f651c009b80 tx=0x7f651c0092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:53.871+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6524013070 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.097+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6510005620 con 0x7f65300745c0 2024-03-24T00:28:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.097+0000 7f652e000700 1 -- 172.21.15.123:0/3295868760 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f6524030090 con 0x7f65300745c0 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stdout:# minimal ceph.conf for 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: mon_host = [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 msgr2=0x7f651803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 0x7f651803b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f651c009b80 tx=0x7f651c0092c0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 msgr2=0x7f653019f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 0x7f653019f7e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6524005950 tx=0x7f6524004f80 comp rx=0 tx=0).stop 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 shutdown_connections 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6518039450 0x7f651803b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6530199ee0 0x7f653019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65300745c0 0x7f653019f7e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 --2- 172.21.15.123:0/3295868760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653019fd20 0x7f65301999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.100+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 >> 172.21.15.123:0/3295868760 conn(0x7f65300fd660 msgr2=0x7f6530105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.101+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 shutdown_connections 2024-03-24T00:28:54.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:28:54.101+0000 7f6538400700 1 -- 172.21.15.123:0/3295868760 wait complete. 2024-03-24T00:28:54.665 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3295868760' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3295868760' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3295868760' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:55.429 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T00:28:55.429 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:28:55.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:28:55.527 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:28:55.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:55.659 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:28:55.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:28:55.689 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:28:55.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:55.755 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:28:55.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:28:55.787 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:28:55.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:28:55.856 INFO:tasks.cephadm:Adding mgr.x on smithi123 2024-03-24T00:28:55.856 INFO:tasks.cephadm:Adding mgr.y on smithi177 2024-03-24T00:28:55.856 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-03-24T00:28:55.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi123=x;smithi177=y;smithi204=z' 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: Reconfiguring daemon mon.b on smithi177 2024-03-24T00:28:56.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:55 smithi204 ceph-mon[36345]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:56.128 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:28:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:28:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: Reconfiguring daemon mon.b on smithi177 2024-03-24T00:28:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:55 smithi177 ceph-mon[36034]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:28:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:56.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:56.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:56.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: Reconfiguring daemon mon.b on smithi177 2024-03-24T00:28:56.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:55 smithi123 ceph-mon[31487]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: Reconfiguring daemon mon.c on smithi204 2024-03-24T00:28:57.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:57 smithi204 ceph-mon[36345]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: Reconfiguring daemon mon.c on smithi204 2024-03-24T00:28:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:57 smithi123 ceph-mon[31487]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:28:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:28:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:28:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: Reconfiguring daemon mon.c on smithi204 2024-03-24T00:28:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:57 smithi177 ceph-mon[36034]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 -- 172.21.15.204:0/2364319743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01026a0 msgr2=0x7fc8d0102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 --2- 172.21.15.204:0/2364319743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01026a0 0x7fc8d0102ac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00099c0 tx=0x7fc8c0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 -- 172.21.15.204:0/2364319743 shutdown_connections 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 --2- 172.21.15.204:0/2364319743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0101d80 0x7fc8d0109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 --2- 172.21.15.204:0/2364319743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01026a0 0x7fc8d0102ac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 --2- 172.21.15.204:0/2364319743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8d01013e0 0x7fc8d0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.565 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 -- 172.21.15.204:0/2364319743 >> 172.21.15.204:0/2364319743 conn(0x7fc8d00fbe30 msgr2=0x7fc8d00fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:58.565 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 -- 172.21.15.204:0/2364319743 shutdown_connections 2024-03-24T00:28:58.565 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.564+0000 7fc8d653c700 1 -- 172.21.15.204:0/2364319743 wait complete. 2024-03-24T00:28:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 Processor -- start 2024-03-24T00:28:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 -- start start 2024-03-24T00:28:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8d0101d80 0x7fc8d01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:58.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 0x7fc8d019fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d010cd80 con 0x7fc8d0101d80 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d010cc00 con 0x7fc8d01026a0 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d653c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d010cf00 con 0x7fc8d01013e0 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:49008/0 (socket says 172.21.15.204:49008) 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 -- 172.21.15.204:0/3538969791 learned_addr learned my addr 172.21.15.204:0/3538969791 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d4ad9700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 0x7fc8d019fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 -- 172.21.15.204:0/3538969791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 msgr2=0x7fc8d019fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 0x7fc8d019fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 -- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8d0101d80 msgr2=0x7fc8d01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8d0101d80 0x7fc8d01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8cffff700 1 -- 172.21.15.204:0/3538969791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d01a00c0 con 0x7fc8d01013e0 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.565+0000 7fc8d4ad9700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 0x7fc8d019fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:28:58.568 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8cffff700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0005e80 tx=0x7fc8c0003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0027070 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8c0022470 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d01a0350 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000f6c0 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.566+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d01a0900 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.567+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc8c002a070 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.567+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d0068e50 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.567+0000 7fc8cd7fa700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 0x7fc8b803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.567+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc8c004d070 con 0x7fc8d01013e0 2024-03-24T00:28:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.568+0000 7fc8cf7fe700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 0x7fc8b803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:28:58.570 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.568+0000 7fc8cf7fe700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 0x7fc8b803b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40097b0 tx=0x7fc8c4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:28:58.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.572+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc8c0013070 con 0x7fc8d01013e0 2024-03-24T00:28:58.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.778+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi123=x;smithi177=y;smithi204=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8d01009e0 con 0x7fc8b8039200 2024-03-24T00:28:58.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.791+0000 7fc8cd7fa700 1 -- 172.21.15.204:0/3538969791 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc8d01009e0 con 0x7fc8b8039200 2024-03-24T00:28:58.791 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 msgr2=0x7fc8b803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 0x7fc8b803b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40097b0 tx=0x7fc8c4006d20 comp rx=0 tx=0).stop 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 msgr2=0x7fc8d0198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0005e80 tx=0x7fc8c0003680 comp rx=0 tx=0).stop 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 shutdown_connections 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc8b8039200 0x7fc8b803b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d01026a0 0x7fc8d019fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.793 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01013e0 0x7fc8d0198460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 --2- 172.21.15.204:0/3538969791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8d0101d80 0x7fc8d01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:28:58.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 >> 172.21.15.204:0/3538969791 conn(0x7fc8d00fbe30 msgr2=0x7fc8d01002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:28:58.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 shutdown_connections 2024-03-24T00:28:58.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:28:58.793+0000 7fc8d653c700 1 -- 172.21.15.204:0/3538969791 wait complete. 2024-03-24T00:28:59.317 DEBUG:teuthology.orchestra.run.smithi177:mgr.y> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T00:28:59.320 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T00:28:59.323 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T00:28:59.323 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:28:59.323 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:28:59.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:28:59.378 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-03-24T00:28:59.429 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:28:59.429 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:28:59.429 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 639 Links: 1 2024-03-24T00:28:59.429 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:28:59.429 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:28:59.430 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-24 00:27:57.204629300 +0000 2024-03-24T00:28:59.430 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-24 00:22:50.118398765 +0000 2024-03-24T00:28:59.430 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-24 00:22:50.118398765 +0000 2024-03-24T00:28:59.430 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-24 00:22:50.118398765 +0000 2024-03-24T00:28:59.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:28:59.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:28:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:28:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.481 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:28:59.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:28:59.483 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:28:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.509 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.509 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.509 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:28:59.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:28:59.511 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:28:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:59.518 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-24T00:28:59.518 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-24T00:28:59.518 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00034417 s, 1.5 MB/s 2024-03-24T00:28:59.523 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:28:59.580 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 656 Links: 1 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:28:59.631 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-24 00:27:57.276627604 +0000 2024-03-24T00:28:59.632 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-24 00:22:50.663385609 +0000 2024-03-24T00:28:59.632 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-24 00:22:50.663385609 +0000 2024-03-24T00:28:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-24 00:22:50.663385609 +0000 2024-03-24T00:28:59.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:28:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-24T00:28:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-24T00:28:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000328054 s, 1.6 MB/s 2024-03-24T00:28:59.726 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:28:59.780 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-03-24T00:28:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:28:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:28:59.828 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 678 Links: 1 2024-03-24T00:28:59.828 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:28:59.829 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:28:59.829 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-24 00:27:57.327626403 +0000 2024-03-24T00:28:59.829 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-24 00:22:51.233371849 +0000 2024-03-24T00:28:59.829 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-24 00:22:51.233371849 +0000 2024-03-24T00:28:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-24 00:22:51.233371849 +0000 2024-03-24T00:28:59.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:28:59.916 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-24T00:28:59.916 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-24T00:28:59.917 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000325391 s, 1.6 MB/s 2024-03-24T00:28:59.920 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:28:59.970 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-03-24T00:29:00.018 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:29:00.018 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.018 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-03-24T00:29:00.018 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.018 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.019 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-24 00:27:57.378625201 +0000 2024-03-24T00:29:00.019 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-24 00:22:51.792358355 +0000 2024-03-24T00:29:00.019 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-24 00:22:51.792358355 +0000 2024-03-24T00:29:00.019 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-24 00:22:51.792358355 +0000 2024-03-24T00:29:00.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:29:00.102 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-24T00:29:00.102 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-24T00:29:00.103 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000315879 s, 1.6 MB/s 2024-03-24T00:29:00.107 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:29:00.162 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-24T00:29:00.162 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:29:00.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:29:00.181 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-24 00:28:43.089712172 +0000 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-24 00:21:41.841410916 +0000 2024-03-24T00:29:00.241 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-24 00:21:41.841410916 +0000 2024-03-24T00:29:00.242 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-24 00:21:41.841410916 +0000 2024-03-24T00:29:00.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:29:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-24T00:29:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-24T00:29:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000227319 s, 2.3 MB/s 2024-03-24T00:29:00.311 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:29:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:00 smithi123 ceph-mon[31487]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi123=x;smithi177=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:00 smithi123 ceph-mon[31487]: Saving service mgr spec with placement smithi123=x;smithi177=y;smithi204=z;count:3 2024-03-24T00:29:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:00 smithi123 ceph-mon[31487]: Deploying daemon mgr.z on smithi204 2024-03-24T00:29:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:00 smithi123 ceph-mon[31487]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:00.368 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-24T00:29:00.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:00 smithi204 ceph-mon[36345]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi123=x;smithi177=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:00.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:00 smithi204 ceph-mon[36345]: Saving service mgr spec with placement smithi123=x;smithi177=y;smithi204=z;count:3 2024-03-24T00:29:00.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:00 smithi204 ceph-mon[36345]: Deploying daemon mgr.z on smithi204 2024-03-24T00:29:00.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:00 smithi204 ceph-mon[36345]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-24 00:28:43.090712145 +0000 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-24 00:21:42.194401028 +0000 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-24 00:21:42.194401028 +0000 2024-03-24T00:29:00.425 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-24 00:21:42.194401028 +0000 2024-03-24T00:29:00.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:29:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:00 smithi177 ceph-mon[36034]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi123=x;smithi177=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:00 smithi177 ceph-mon[36034]: Saving service mgr spec with placement smithi123=x;smithi177=y;smithi204=z;count:3 2024-03-24T00:29:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:00 smithi177 ceph-mon[36034]: Deploying daemon mgr.z on smithi204 2024-03-24T00:29:00.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:00 smithi177 ceph-mon[36034]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:00.489 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-24T00:29:00.489 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-24T00:29:00.489 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000228118 s, 2.2 MB/s 2024-03-24T00:29:00.490 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:29:00.547 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-24T00:29:00.603 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-24 00:28:43.090712145 +0000 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-24 00:21:42.530391617 +0000 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-24 00:21:42.530391617 +0000 2024-03-24T00:29:00.604 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-24 00:21:42.530391617 +0000 2024-03-24T00:29:00.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:29:00.670 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-24T00:29:00.671 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-24T00:29:00.671 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000181496 s, 2.8 MB/s 2024-03-24T00:29:00.672 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:29:00.728 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-24T00:29:00.757 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:00 smithi204 systemd[1]: Starting Ceph mgr.z for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:29:00.784 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-24 00:28:43.090712145 +0000 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-24 00:21:42.862382317 +0000 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-24 00:21:42.862382317 +0000 2024-03-24T00:29:00.785 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-24 00:21:42.862382317 +0000 2024-03-24T00:29:00.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:29:00.851 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-24T00:29:00.851 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-24T00:29:00.851 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00017151 s, 3.0 MB/s 2024-03-24T00:29:00.852 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:29:00.908 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-24T00:29:00.908 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:29:00.924 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:29:00.925 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-24T00:29:00.979 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:29:00.979 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:00.979 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-24T00:29:00.979 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:00.980 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:00.980 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-24 00:29:00.693124986 +0000 2024-03-24T00:29:00.980 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-24 00:21:48.276358103 +0000 2024-03-24T00:29:00.980 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-24 00:21:48.276358103 +0000 2024-03-24T00:29:00.980 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-24 00:21:48.276358103 +0000 2024-03-24T00:29:00.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:29:01.044 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:00 smithi204 podman[38035]: 2024-03-24 00:29:00.876396591 +0000 UTC m=+0.017495931 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:29:01.044 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:00 smithi204 podman[38035]: 2024-03-24 00:29:00.998274178 +0000 UTC m=+0.139373513 container create d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T00:29:01.047 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-24T00:29:01.047 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-24T00:29:01.047 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00033658 s, 1.5 MB/s 2024-03-24T00:29:01.049 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:29:01.105 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-24T00:29:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-24 00:29:00.693124986 +0000 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-24 00:21:48.638353352 +0000 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-24 00:21:48.638353352 +0000 2024-03-24T00:29:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-24 00:21:48.638353352 +0000 2024-03-24T00:29:01.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:29:01.224 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-24T00:29:01.224 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-24T00:29:01.224 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00024466 s, 2.1 MB/s 2024-03-24T00:29:01.225 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:29:01.281 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-24 00:29:00.694124973 +0000 2024-03-24T00:29:01.337 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-24 00:21:48.964349074 +0000 2024-03-24T00:29:01.338 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-24 00:21:48.964349074 +0000 2024-03-24T00:29:01.338 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-24 00:21:48.964349074 +0000 2024-03-24T00:29:01.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:29:01.404 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-24T00:29:01.404 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-24T00:29:01.404 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00022372 s, 2.3 MB/s 2024-03-24T00:29:01.405 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:29:01.461 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-24 00:29:00.694124973 +0000 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-24 00:21:49.291344783 +0000 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-24 00:21:49.291344783 +0000 2024-03-24T00:29:01.517 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-24 00:21:49.291344783 +0000 2024-03-24T00:29:01.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:29:01.580 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 podman[38035]: 2024-03-24 00:29:01.325787887 +0000 UTC m=+0.466887206 container init d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:29:01.580 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 podman[38035]: 2024-03-24 00:29:01.329230822 +0000 UTC m=+0.470330135 container start d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T00:29:01.581 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 ceph-mgr[38181]: -- 172.21.15.204:0/3096953596 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5613dc99e5a0 con 0x5613dc973000 2024-03-24T00:29:01.581 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 bash[38035]: d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 2024-03-24T00:29:01.581 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 systemd[1]: Started Ceph mgr.z for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:29:01.583 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-24T00:29:01.583 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-24T00:29:01.583 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000217326 s, 2.4 MB/s 2024-03-24T00:29:01.585 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:29:01.640 INFO:tasks.cephadm:Deploying osd.0 on smithi123 with /dev/vg_nvme/lv_4... 2024-03-24T00:29:01.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:29:02.068 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:29:02.143 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:01.891+0000 7ff4653ac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:29:02.143 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:01.985+0000 7ff4653ac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:02.400 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:02.449 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:02.450 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:02.959 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:02.628+0000 7ff4653ac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:29:02.959 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:02.722+0000 7ff4653ac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:29:02.959 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:02.864+0000 7ff4653ac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:29:03.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-mon[36345]: Deploying daemon mgr.y on smithi177 2024-03-24T00:29:03.306 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:02.958+0000 7ff4653ac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:29:03.306 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.041+0000 7ff4653ac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:29:03.306 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.305+0000 7ff4653ac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:29:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:03 smithi123 ceph-mon[31487]: Deploying daemon mgr.y on smithi177 2024-03-24T00:29:03.457 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:03 smithi177 ceph-mon[36034]: Deploying daemon mgr.y on smithi177 2024-03-24T00:29:03.571 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.414+0000 7ff4653ac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:29:03.571 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.500+0000 7ff4653ac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:29:03.712 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:03 smithi177 systemd[1]: Starting Ceph mgr.y for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:29:04.071 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.594+0000 7ff4653ac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:29:04.071 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:03.771+0000 7ff4653ac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:03 smithi177 podman[37443]: 2024-03-24 00:29:03.830143914 +0000 UTC m=+0.104457678 container create 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:03 smithi177 podman[37443]: 2024-03-24 00:29:03.743234474 +0000 UTC m=+0.017548227 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 podman[37443]: 2024-03-24 00:29:04.007656845 +0000 UTC m=+0.281970606 container init 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 podman[37443]: 2024-03-24 00:29:04.010097501 +0000 UTC m=+0.284411238 container start 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 ceph-mgr[37462]: -- 172.21.15.177:0/3020107389 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f22ef5a5a0 con 0x55f22ef2f000 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 bash[37443]: 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 2024-03-24T00:29:04.158 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 systemd[1]: Started Ceph mgr.y for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:29:04.348 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:04 smithi123 ceph-mon[31487]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:04 smithi177 ceph-mon[36034]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:04 smithi204 ceph-mon[36345]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0365032 s, 287 MB/s 2024-03-24T00:29:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-24T00:29:04.962 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:04.562+0000 7f588cdc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:29:04.962 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:04.659+0000 7f588cdc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:29:05.071 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:04.769+0000 7ff4653ac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:29:05.071 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:04.905+0000 7ff4653ac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:05.275+0000 7ff4653ac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:05.368+0000 7ff4653ac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.416 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:05.446 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:05.344+0000 7f588cdc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:05.446+0000 7f588cdc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:05.571 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:05 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.571 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:05 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.571 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:05 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.571 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:05 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:05.572 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:05 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:05.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-03-24T00:29:05.712 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:05.587+0000 7f588cdc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:29:05.712 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:05.689+0000 7f588cdc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:29:05.760 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:05.568+0000 7ff4653ac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:29:05.760 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:05.760+0000 7ff4653ac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:29:06.045 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:05.775+0000 7f588cdc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:29:06.046 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:06.045+0000 7f588cdc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:29:06.080 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:06.047+0000 7ff4653ac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:06.308+0000 7ff4653ac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.310 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:06.310 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:06.310 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:06.154+0000 7f588cdc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:06.239+0000 7f588cdc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:06.331+0000 7f588cdc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:06.332 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.333 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.333 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:06.333 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:06.333 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:29:06.423 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:29:06.424 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:06.571 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:06.544+0000 7ff4653ac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:29:06.712 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:06.507+0000 7f588cdc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:29:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:07 smithi123 ceph-mon[31487]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:29:07.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:07 smithi123 ceph-mon[31487]: Reconfiguring daemon mgr.x on smithi123 2024-03-24T00:29:07.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:07 smithi177 ceph-mon[36034]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:29:07.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:07 smithi177 ceph-mon[36034]: Reconfiguring daemon mgr.x on smithi123 2024-03-24T00:29:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:07 smithi204 ceph-mon[36345]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:29:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:07 smithi204 ceph-mon[36345]: Reconfiguring daemon mgr.x on smithi123 2024-03-24T00:29:07.571 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:07.339+0000 7ff4653ac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:29:07.933 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:07.472+0000 7f588cdc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:29:07.933 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:07.600+0000 7f588cdc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:29:07.933 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:07.933+0000 7f588cdc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:29:08.171 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:07.912+0000 7ff4653ac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:29:08.171 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:08.008+0000 7ff4653ac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:29:08.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-mon[36034]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:08.212 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:08.021+0000 7f588cdc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:29:08.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 -- 172.21.15.123:0/3330112014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 msgr2=0x7f9bb0103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:08.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 --2- 172.21.15.123:0/3330112014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb0103830 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009990 tx=0x7f9b9c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.238 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:08 smithi123 ceph-mon[31487]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:08.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 -- 172.21.15.123:0/3330112014 shutdown_connections 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 --2- 172.21.15.123:0/3330112014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb0103d70 0x7f9bb010a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 --2- 172.21.15.123:0/3330112014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb01021d0 0x7f9bb01025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 --2- 172.21.15.123:0/3330112014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb0103830 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 -- 172.21.15.123:0/3330112014 >> 172.21.15.123:0/3330112014 conn(0x7f9bb00fd730 msgr2=0x7f9bb00ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 -- 172.21.15.123:0/3330112014 shutdown_connections 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.236+0000 7f9bb5800700 1 -- 172.21.15.123:0/3330112014 wait complete. 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.238+0000 7f9bb5800700 1 Processor -- start 2024-03-24T00:29:08.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.238+0000 7f9bb5800700 1 -- start start 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.238+0000 7f9bb5800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb01021d0 0x7f9bb019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bb5800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bb5800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 0x7f9bb01a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bb5800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb010deb0 con 0x7f9bb0103d70 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bb5800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb010dd30 con 0x7f9bb01021d0 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bb5800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb010e030 con 0x7f9bb01033d0 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:08.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9baea00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb01021d0 0x7f9bb019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50660/0 (socket says 172.21.15.123:50660) 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 -- 172.21.15.123:0/4187466783 learned_addr learned my addr 172.21.15.123:0/4187466783 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9baf400700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 0x7f9bb01a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 -- 172.21.15.123:0/4187466783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb01021d0 msgr2=0x7f9bb019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb01021d0 0x7f9bb019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 -- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 msgr2=0x7f9bb01a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:08.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 0x7f9bb01a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:08.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9bae000700 1 -- 172.21.15.123:0/4187466783 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb01a2330 con 0x7f9bb01033d0 2024-03-24T00:29:08.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.239+0000 7f9baf400700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 0x7f9bb01a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:29:08.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9bae000700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00f690 tx=0x7f9b9c00f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:29:08.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c026070 con 0x7f9bb01033d0 2024-03-24T00:29:08.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb01a25c0 con 0x7f9bb01033d0 2024-03-24T00:29:08.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c0218f0 con 0x7f9bb01033d0 2024-03-24T00:29:08.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c017650 con 0x7f9bb01033d0 2024-03-24T00:29:08.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.240+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb01a2aa0 con 0x7f9bb01033d0 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.241+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f9b9c029070 con 0x7f9bb01033d0 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.241+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb0068e50 con 0x7f9bb01033d0 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.241+0000 7f9ba7400700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 0x7f9b9403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.242+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9b9c04d0c0 con 0x7f9bb01033d0 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.242+0000 7f9baea00700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 0x7f9b9403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:08.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.242+0000 7f9baea00700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 0x7f9b9403b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0103230 tx=0x7f9ba0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:29:08.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.251+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b9c013070 con 0x7f9bb01033d0 2024-03-24T00:29:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:08.488+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bb0075ae0 con 0x7f9b94039200 2024-03-24T00:29:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:08 smithi204 ceph-mon[36345]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:08.571 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:29:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z[38120]: 2024-03-24T00:29:08.221+0000 7ff4653ac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:29:08.676 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:08.215+0000 7f588cdc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:29:08.677 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:08.398+0000 7f588cdc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:29:08.938 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:08.676+0000 7f588cdc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:29:08.938 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:08.937+0000 7f588cdc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:29:09.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:09.182+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f9b9c009d10 con 0x7f9bb01033d0 2024-03-24T00:29:09.212 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:09.171+0000 7f588cdc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:29:09.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:29:09.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:09.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: Standby manager daemon z started 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: Standby manager daemon z started 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:09.539 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.539 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.539 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: Standby manager daemon z started 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.204:0/3091108398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:10.406 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:10 smithi123 ceph-mon[31487]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:10.406 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:10 smithi123 ceph-mon[31487]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:10.406 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:10 smithi123 ceph-mon[31487]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T00:29:10.406 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:29:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-mon[36034]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-mon[36034]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-mon[36034]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T00:29:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:29:10.463 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:09.966+0000 7f588cdc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:29:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:10 smithi204 ceph-mon[36345]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:10 smithi204 ceph-mon[36345]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:10 smithi204 ceph-mon[36345]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T00:29:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:29:10.837 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:10.532+0000 7f588cdc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:29:10.838 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:10.626+0000 7f588cdc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:29:10.838 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:29:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y[37458]: 2024-03-24T00:29:10.836+0000 7f588cdc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:29:11.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:11.205+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mon.1 v2:172.21.15.204:3300/0 8 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f9b9c0177f0 con 0x7f9bb01033d0 2024-03-24T00:29:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:11 smithi177 ceph-mon[36034]: Standby manager daemon y started 2024-03-24T00:29:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:11 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:29:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:11 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:11 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:29:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:11 smithi177 ceph-mon[36034]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:11.560 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:11 smithi123 ceph-mon[31487]: Standby manager daemon y started 2024-03-24T00:29:11.560 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:11 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:29:11.560 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:11 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:11.560 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:11 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:29:11.560 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:11 smithi123 ceph-mon[31487]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:11 smithi204 ceph-mon[36345]: Standby manager daemon y started 2024-03-24T00:29:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:11 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:29:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:11 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:29:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:11 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:29:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:11 smithi204 ceph-mon[36345]: from='mgr.? 172.21.15.177:0/896117033' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:29:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:12 smithi177 ceph-mon[36034]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:12 smithi177 ceph-mon[36034]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:29:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:29:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:12 smithi204 ceph-mon[36345]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:12 smithi204 ceph-mon[36345]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:29:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:29:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:12 smithi123 ceph-mon[31487]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:12 smithi123 ceph-mon[31487]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:29:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:29:13.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:13 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]: dispatch 2024-03-24T00:29:13.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:13 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]': finished 2024-03-24T00:29:13.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:13 smithi177 ceph-mon[36034]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:29:13.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:13.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:13 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]: dispatch 2024-03-24T00:29:13.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:13 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]': finished 2024-03-24T00:29:13.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:13 smithi204 ceph-mon[36345]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:29:13.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:13.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:13 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]: dispatch 2024-03-24T00:29:13.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:13 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2549664216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bfeabab-d57d-4786-9124-62355d56818c"}]': finished 2024-03-24T00:29:13.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:13 smithi123 ceph-mon[31487]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:29:13.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:14.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:14 smithi177 ceph-mon[36034]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:14 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2371794580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:14 smithi204 ceph-mon[36345]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:14 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2371794580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:14 smithi123 ceph-mon[31487]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:14 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2371794580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:16 smithi177 ceph-mon[36034]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:16 smithi204 ceph-mon[36345]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:16 smithi123 ceph-mon[31487]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:18 smithi204 ceph-mon[36345]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:18 smithi123 ceph-mon[31487]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:18 smithi177 ceph-mon[36034]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:20.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:20 smithi123 ceph-mon[31487]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:20 smithi204 ceph-mon[36345]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:20 smithi177 ceph-mon[36034]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:21.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:21 smithi123 ceph-mon[31487]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:21.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:21 smithi204 ceph-mon[36345]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:21.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:21 smithi177 ceph-mon[36034]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:29:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:22 smithi204 ceph-mon[36345]: Deploying daemon osd.0 on smithi123 2024-03-24T00:29:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:29:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:22 smithi123 ceph-mon[31487]: Deploying daemon osd.0 on smithi123 2024-03-24T00:29:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:29:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:22.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:22 smithi177 ceph-mon[36034]: Deploying daemon osd.0 on smithi123 2024-03-24T00:29:23.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:23 smithi204 ceph-mon[36345]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:23.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:23 smithi123 ceph-mon[31487]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:23.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:23 smithi177 ceph-mon[36034]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:26.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:26 smithi204 ceph-mon[36345]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:26.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:26 smithi123 ceph-mon[31487]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:26 smithi177 ceph-mon[36034]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:28 smithi204 ceph-mon[36345]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:28.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:28 smithi123 ceph-mon[31487]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:28 smithi177 ceph-mon[36034]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:30 smithi204 ceph-mon[36345]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:30 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:30 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:30 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:30 smithi123 ceph-mon[31487]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:30 smithi177 ceph-mon[36034]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:32.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:32 smithi123 ceph-mon[31487]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:32.275 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:32 smithi177 ceph-mon[36034]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:32.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:32 smithi204 ceph-mon[36345]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:34.346 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:34.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:34 smithi123 ceph-mon[31487]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:34.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:34 smithi177 ceph-mon[36034]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:34.478 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 0 on host 'smithi123' 2024-03-24T00:29:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.477+0000 7f9ba7400700 1 -- 172.21.15.123:0/4187466783 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9bb0075ae0 con 0x7f9b94039200 2024-03-24T00:29:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 msgr2=0x7f9b9403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 0x7f9b9403b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0103230 tx=0x7f9ba0009190 comp rx=0 tx=0).stop 2024-03-24T00:29:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 msgr2=0x7f9bb019ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00f690 tx=0x7f9b9c00f6c0 comp rx=0 tx=0).stop 2024-03-24T00:29:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 shutdown_connections 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9b94039200 0x7f9b9403b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.481+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bb0103d70 0x7f9bb01a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.482+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bb01021d0 0x7f9bb019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.482+0000 7f9bb5800700 1 --2- 172.21.15.123:0/4187466783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bb01033d0 0x7f9bb019ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.482+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 >> 172.21.15.123:0/4187466783 conn(0x7f9bb00fd730 msgr2=0x7f9bb00743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.482+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 shutdown_connections 2024-03-24T00:29:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:29:34.482+0000 7f9bb5800700 1 -- 172.21.15.123:0/4187466783 wait complete. 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:34.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:34 smithi204 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:29:35.219 DEBUG:teuthology.orchestra.run.smithi123:osd.0> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T00:29:35.223 INFO:tasks.cephadm:Deploying osd.1 on smithi177 with /dev/vg_nvme/lv_4... 2024-03-24T00:29:35.223 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:29:35.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:29:35.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:29:35.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:35.454 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.454 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.454 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.454 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:35.471 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:36 smithi177 ceph-mon[36034]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:36 smithi177 ceph-mon[36034]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:29:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:36 smithi177 ceph-mon[36034]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:29:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:36.485 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:36 smithi123 ceph-mon[31487]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:36.485 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:36 smithi123 ceph-mon[31487]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:29:36.485 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:36 smithi123 ceph-mon[31487]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:29:36.486 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:36.486 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:36 smithi204 ceph-mon[36345]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:36 smithi204 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:29:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:36 smithi204 ceph-mon[36345]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:29:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:37.438 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:29:37.439 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:29:37.439 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-24T00:29:37.439 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-24T00:29:37.439 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111487 s, 941 MB/s 2024-03-24T00:29:37.439 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-24T00:29:37.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:37 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:29:37.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:37 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:29:37.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:37 smithi177 ceph-mon[36034]: osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] boot 2024-03-24T00:29:37.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:37 smithi177 ceph-mon[36034]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:29:37.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:37 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:37 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:29:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:37 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:29:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:37 smithi204 ceph-mon[36345]: osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] boot 2024-03-24T00:29:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:37 smithi204 ceph-mon[36345]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:29:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:37 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:37.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:37 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:29:37.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:37 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:29:37.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:37 smithi123 ceph-mon[31487]: osd.0 [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] boot 2024-03-24T00:29:37.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:37 smithi123 ceph-mon[31487]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:29:37.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:37 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:29:38.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-03-24T00:29:38.267 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:29:38.388 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:38 smithi177 ceph-mon[36034]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:38.388 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:38 smithi177 ceph-mon[36034]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:29:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:38 smithi204 ceph-mon[36345]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:38 smithi204 ceph-mon[36345]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:29:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:38 smithi123 ceph-mon[31487]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:29:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:38 smithi123 ceph-mon[31487]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:29:39.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 -- 172.21.15.177:0/252157329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1021c0 msgr2=0x7fd30c1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 --2- 172.21.15.177:0/252157329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1021c0 0x7fd30c1025e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0099c0 tx=0x7fd2fc009cd0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 -- 172.21.15.177:0/252157329 shutdown_connections 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 --2- 172.21.15.177:0/252157329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 0x7fd30c10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 --2- 172.21.15.177:0/252157329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1021c0 0x7fd30c1025e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 --2- 172.21.15.177:0/252157329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd30c1033c0 0x7fd30c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 -- 172.21.15.177:0/252157329 >> 172.21.15.177:0/252157329 conn(0x7fd30c0fd740 msgr2=0x7fd30c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 -- 172.21.15.177:0/252157329 shutdown_connections 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.864+0000 7fd311160700 1 -- 172.21.15.177:0/252157329 wait complete. 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 Processor -- start 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 -- start start 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd30c1021c0 0x7fd30c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:39.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 0x7fd30c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c10de30 con 0x7fd30c1021c0 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c10dcb0 con 0x7fd30c1033c0 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd311160700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c10dfb0 con 0x7fd30c103d60 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:37796/0 (socket says 172.21.15.177:37796) 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 -- 172.21.15.177:0/2816797087 learned_addr learned my addr 172.21.15.177:0/2816797087 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd30b59e700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 0x7fd30c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 -- 172.21.15.177:0/2816797087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 msgr2=0x7fd30c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 0x7fd30c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 -- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd30c1021c0 msgr2=0x7fd30c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd30c1021c0 0x7fd30c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 -- 172.21.15.177:0/2816797087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c1a00b0 con 0x7fd30c1033c0 2024-03-24T00:29:39.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.865+0000 7fd303fff700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b530 tx=0x7fd2f400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:29:39.869 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.866+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4015020 con 0x7fd30c1033c0 2024-03-24T00:29:39.869 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.866+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c1a03a0 con 0x7fd30c1033c0 2024-03-24T00:29:39.869 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.866+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4011d10 con 0x7fd30c1033c0 2024-03-24T00:29:39.869 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.866+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400f2e0 con 0x7fd30c1033c0 2024-03-24T00:29:39.869 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.866+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c1a0960 con 0x7fd30c1033c0 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.867+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c068e50 con 0x7fd30c1033c0 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.868+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7fd2f4018020 con 0x7fd30c1033c0 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.868+0000 7fd308d99700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 0x7fd2f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.869+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fd2f40ccd40 con 0x7fd30c1033c0 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.869+0000 7fd30ad9d700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 0x7fd2f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:29:39.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.870+0000 7fd30ad9d700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 0x7fd2f80a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc003890 tx=0x7fd2fc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:29:39.872 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:39.872+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2f401f020 con 0x7fd30c1033c0 2024-03-24T00:29:40.077 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:40.077+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd30c0625d0 con 0x7fd2f80a2800 2024-03-24T00:29:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:29:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:29:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:29:40.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:41.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:41 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi123 2024-03-24T00:29:41.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:41 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi123 to 17091M 2024-03-24T00:29:41.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:41 smithi177 ceph-mon[36034]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:41.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:41 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi123 2024-03-24T00:29:41.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:41 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi123 to 17091M 2024-03-24T00:29:41.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:41 smithi123 ceph-mon[31487]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:41 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi123 2024-03-24T00:29:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:41 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi123 to 17091M 2024-03-24T00:29:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:41 smithi204 ceph-mon[36345]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:29:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:42 smithi177 ceph-mon[36034]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:42 smithi204 ceph-mon[36345]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:42 smithi123 ceph-mon[31487]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/3842868516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:43 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:43 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]': finished 2024-03-24T00:29:43.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:43 smithi177 ceph-mon[36034]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:29:43.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:29:43.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:43 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/3842868516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:43 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:43 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]': finished 2024-03-24T00:29:43.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:43 smithi123 ceph-mon[31487]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:29:43.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:29:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:43 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/3842868516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:43 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]: dispatch 2024-03-24T00:29:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:43 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97ec847a-50c8-46e1-bee7-f7ee7a409894"}]': finished 2024-03-24T00:29:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:43 smithi204 ceph-mon[36345]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:29:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:29:44.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:44 smithi177 ceph-mon[36034]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:44.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:44 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/3168765213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:44 smithi204 ceph-mon[36345]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:44 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/3168765213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:44 smithi123 ceph-mon[31487]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:44 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/3168765213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:29:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:46 smithi177 ceph-mon[36034]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:46 smithi204 ceph-mon[36345]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:46 smithi123 ceph-mon[31487]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:48 smithi177 ceph-mon[36034]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:48 smithi204 ceph-mon[36345]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:48 smithi123 ceph-mon[31487]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:50 smithi177 ceph-mon[36034]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:29:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:50 smithi204 ceph-mon[36345]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:29:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:50 smithi123 ceph-mon[31487]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:29:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:51.473 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:51 smithi177 ceph-mon[36034]: Deploying daemon osd.1 on smithi177 2024-03-24T00:29:51.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:51 smithi204 ceph-mon[36345]: Deploying daemon osd.1 on smithi177 2024-03-24T00:29:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:51 smithi123 ceph-mon[31487]: Deploying daemon osd.1 on smithi177 2024-03-24T00:29:52.473 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:52 smithi177 ceph-mon[36034]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:52.556 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:52 smithi123 ceph-mon[31487]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:52 smithi204 ceph-mon[36345]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:54.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:54 smithi177 ceph-mon[36034]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:54 smithi204 ceph-mon[36345]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:54 smithi123 ceph-mon[31487]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:56 smithi123 ceph-mon[31487]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:56 smithi177 ceph-mon[36034]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:56 smithi204 ceph-mon[36345]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:58.116 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:58 smithi177 ceph-mon[36034]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:58.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.213+0000 7fd308d99700 1 -- 172.21.15.177:0/2816797087 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd30c0625d0 con 0x7fd2f80a2800 2024-03-24T00:29:58.214 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 1 on host 'smithi177' 2024-03-24T00:29:58.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 msgr2=0x7fd2f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:58.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 0x7fd2f80a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc003890 tx=0x7fd2fc000bc0 comp rx=0 tx=0).stop 2024-03-24T00:29:58.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 msgr2=0x7fd30c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:29:58.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b530 tx=0x7fd2f400b840 comp rx=0 tx=0).stop 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 shutdown_connections 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd2f80a2800 0x7fd2f80a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd30c103d60 0x7fd30c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd30c1021c0 0x7fd30c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 --2- 172.21.15.177:0/2816797087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd30c1033c0 0x7fd30c198990 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 >> 172.21.15.177:0/2816797087 conn(0x7fd30c0fd740 msgr2=0x7fd30c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 shutdown_connections 2024-03-24T00:29:58.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:29:58.217+0000 7fd311160700 1 -- 172.21.15.177:0/2816797087 wait complete. 2024-03-24T00:29:58.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:58 smithi123 ceph-mon[31487]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:58 smithi204 ceph-mon[36345]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:29:58.859 DEBUG:teuthology.orchestra.run.smithi177:osd.1> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T00:29:58.861 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/vg_nvme/lv_4... 2024-03-24T00:29:58.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:29:59.019 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:29:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:29:58.972+0000 7f911a1657c0 -1 osd.1 0 log_to_monitors true 2024-03-24T00:29:59.108 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:29:59.313 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.313 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.313 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:29:59.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:29:59 smithi177 ceph-mon[36034]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:29:59.412 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.412 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:29:59.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:29:59 smithi204 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:29:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:29:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:29:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:29:59 smithi123 ceph-mon[31487]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:30:00.256 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:00 smithi204 ceph-mon[36345]: overall HEALTH_OK 2024-03-24T00:30:00.386 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:30:00.301+0000 7f910f7ca700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T00:30:00.386 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:30:00.305+0000 7f910b5bd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:00.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:00 smithi177 ceph-mon[36034]: overall HEALTH_OK 2024-03-24T00:30:00.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:00.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:30:00.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: from='osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:30:00.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:00.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:30:00.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:00 smithi123 ceph-mon[31487]: overall HEALTH_OK 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112141 s, 935 MB/s 2024-03-24T00:30:00.828 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-24T00:30:01.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-03-24T00:30:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:01 smithi204 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:30:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:01 smithi204 ceph-mon[36345]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:30:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:01 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:01 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:01 smithi204 ceph-mon[36345]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:01 smithi123 ceph-mon[31487]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:30:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:01 smithi123 ceph-mon[31487]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:30:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:01 smithi123 ceph-mon[31487]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:01.622 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:01 smithi177 ceph-mon[36034]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:30:01.623 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:01 smithi177 ceph-mon[36034]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:30:01.623 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:01 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.623 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:01 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:01.623 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:01 smithi177 ceph-mon[36034]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:30:01.623 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:30:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:30:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:30:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] boot 2024-03-24T00:30:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:30:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi177 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] boot 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi177 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:30:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:30:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: osd.1 [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] boot 2024-03-24T00:30:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:30:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi177 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- 172.21.15.204:0/2993978336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8100890 msgr2=0x7f0ca8100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2993978336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8100890 0x7f0ca8100cb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800b0a0 tx=0x7f0c9800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- 172.21.15.204:0/2993978336 shutdown_connections 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2993978336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ca8102430 0x7f0ca8108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2993978336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8100890 0x7f0ca8100cb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.142 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2993978336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8101a90 0x7f0ca8101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- 172.21.15.204:0/2993978336 >> 172.21.15.204:0/2993978336 conn(0x7f0ca80fbe30 msgr2=0x7f0ca80fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:03.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- 172.21.15.204:0/2993978336 shutdown_connections 2024-03-24T00:30:03.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- 172.21.15.204:0/2993978336 wait complete. 2024-03-24T00:30:03.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 Processor -- start 2024-03-24T00:30:03.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.142+0000 7f0cad75c700 1 -- start start 2024-03-24T00:30:03.144 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ca8100890 0x7f0ca8104820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 0x7f0ca81033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca810cd80 con 0x7f0ca8100890 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca810cc00 con 0x7f0ca8102430 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca810cf00 con 0x7f0ca8101a90 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:51628/0 (socket says 172.21.15.204:51628) 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 -- 172.21.15.204:0/2693191482 learned_addr learned my addr 172.21.15.204:0/2693191482 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca77fe700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 0x7f0ca81033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 -- 172.21.15.204:0/2693191482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 msgr2=0x7f0ca81033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 0x7f0ca81033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 -- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ca8100890 msgr2=0x7f0ca8104820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ca8100890 0x7f0ca8104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 -- 172.21.15.204:0/2693191482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca8103c60 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca77fe700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 0x7f0ca81033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0ca67fc700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000d980 tx=0x7f0c9000dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90017070 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c90004d10 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c900055c0 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca81a2670 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.143+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca81a2c00 con 0x7f0ca8101a90 2024-03-24T00:30:03.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.144+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca8068e50 con 0x7f0ca8101a90 2024-03-24T00:30:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.145+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f0c90004750 con 0x7f0ca8101a90 2024-03-24T00:30:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.146+0000 7f0c9ffff700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 0x7f0c940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.146+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f0c900cd100 con 0x7f0ca8101a90 2024-03-24T00:30:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.147+0000 7f0ca6ffd700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 0x7f0c940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:03.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.147+0000 7f0ca6ffd700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 0x7f0c940a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009be0 tx=0x7f0c98008fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:03.151 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.151+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0c90020020 con 0x7f0ca8101a90 2024-03-24T00:30:03.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:03.354+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ca80fdfb0 con 0x7f0c940a2880 2024-03-24T00:30:03.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:03 smithi204 ceph-mon[36345]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:30:03.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:03 smithi204 ceph-mon[36345]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:03 smithi177 ceph-mon[36034]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:03 smithi177 ceph-mon[36034]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:03.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:03 smithi123 ceph-mon[31487]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:30:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:03 smithi123 ceph-mon[31487]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:04.988 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:04 smithi204 ceph-mon[36345]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:05.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:04 smithi123 ceph-mon[31487]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:05.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:04 smithi177 ceph-mon[36034]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:06.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:05 smithi204 ceph-mon[36345]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:06.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:05 smithi123 ceph-mon[31487]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:06.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:05 smithi177 ceph-mon[36034]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/747529319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]': finished 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:07.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:06 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/272320970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/747529319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]': finished 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:07.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:06 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/272320970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/747529319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]: dispatch 2024-03-24T00:30:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "408c94aa-43dc-4ca2-82bf-e7c8973d41d7"}]': finished 2024-03-24T00:30:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:30:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:07.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:06 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/272320970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:08.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:07 smithi204 ceph-mon[36345]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:08.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:07 smithi123 ceph-mon[31487]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:08.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:07 smithi177 ceph-mon[36034]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:10 smithi204 ceph-mon[36345]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:10.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:10 smithi123 ceph-mon[31487]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:10 smithi177 ceph-mon[36034]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:12.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:12 smithi204 ceph-mon[36345]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:12 smithi123 ceph-mon[31487]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:12 smithi177 ceph-mon[36034]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:14.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:14 smithi123 ceph-mon[31487]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:14.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:30:14.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:14.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:14 smithi123 ceph-mon[31487]: Deploying daemon osd.2 on smithi204 2024-03-24T00:30:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:14 smithi204 ceph-mon[36345]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:30:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:14.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:14 smithi204 ceph-mon[36345]: Deploying daemon osd.2 on smithi204 2024-03-24T00:30:14.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:14 smithi177 ceph-mon[36034]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:14.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:30:14.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:14.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:14 smithi177 ceph-mon[36034]: Deploying daemon osd.2 on smithi204 2024-03-24T00:30:16.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:16 smithi204 ceph-mon[36345]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:16 smithi123 ceph-mon[31487]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:16 smithi177 ceph-mon[36034]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:18.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:18 smithi204 ceph-mon[36345]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:18.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:18 smithi123 ceph-mon[31487]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:18.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:18 smithi177 ceph-mon[36034]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:19.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:19.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:19.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:19.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:19.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:19.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:20.418 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:20 smithi204 ceph-mon[36345]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:20.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:20 smithi177 ceph-mon[36034]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:20 smithi123 ceph-mon[31487]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:21.588 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.588+0000 7f0c9ffff700 1 -- 172.21.15.204:0/2693191482 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0ca80fdfb0 con 0x7f0c940a2880 2024-03-24T00:30:21.588 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.590+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 msgr2=0x7f0c940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.590+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 0x7f0c940a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009be0 tx=0x7f0c98008fe0 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.590+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 msgr2=0x7f0ca8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.590+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000d980 tx=0x7f0c9000dd40 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 shutdown_connections 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0c940a2880 0x7f0c940a4d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca8102430 0x7f0ca81033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ca8100890 0x7f0ca8104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 --2- 172.21.15.204:0/2693191482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ca8101a90 0x7f0ca8102e70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 >> 172.21.15.204:0/2693191482 conn(0x7f0ca80fbe30 msgr2=0x7f0ca80fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:21.591 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 shutdown_connections 2024-03-24T00:30:21.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:30:21.591+0000 7f0cad75c700 1 -- 172.21.15.204:0/2693191482 wait complete. 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:21 smithi204 ceph-mon[36345]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:21.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:21.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:21.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:21 smithi123 ceph-mon[31487]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:21 smithi177 ceph-mon[36034]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:22.161 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T00:30:22.163 INFO:tasks.cephadm:Deploying osd.3 on smithi123 with /dev/vg_nvme/lv_3... 2024-03-24T00:30:22.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:30:22.607 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:30:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:22 smithi204 ceph-mon[36345]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:22 smithi204 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:22 smithi123 ceph-mon[31487]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:22 smithi123 ceph-mon[31487]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:22.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:22.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:22 smithi177 ceph-mon[36034]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:22 smithi177 ceph-mon[36034]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.809 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:23 smithi123 ceph-mon[31487]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-mon[36345]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:23.822 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2[41552]: 2024-03-24T00:30:23.584+0000 7fbefb6d3700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T00:30:23.822 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2[41552]: 2024-03-24T00:30:23.588+0000 7fbef74c6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:23.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:30:23.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: from='osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:30:23.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:23.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:30:23.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:23 smithi177 ceph-mon[36034]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:24.799 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:24 smithi204 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:30:24.799 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:24 smithi204 ceph-mon[36345]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:30:24.799 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:24 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:24.834 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:24 smithi123 ceph-mon[31487]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:30:24.834 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:24 smithi123 ceph-mon[31487]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:30:24.834 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:24 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:24.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:24 smithi177 ceph-mon[36034]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:30:24.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:24 smithi177 ceph-mon[36034]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:30:24.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:24 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033415 s, 314 MB/s 2024-03-24T00:30:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] boot 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:30:25.719 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:25.720 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] boot 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:25.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: osd.2 [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] boot 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:30:26.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:26.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:26.090 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-03-24T00:30:26.486 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi204 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi204 to 17080M 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:30:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:26.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi204 2024-03-24T00:30:26.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi204 to 17080M 2024-03-24T00:30:26.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:26.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:26.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:26.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:30:26.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi204 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi204 to 17080M 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:26.975 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:30:26.976 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:30:28.105 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:28.105 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:27 smithi123 ceph-mon[31487]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:30:28.105 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:27 smithi123 ceph-mon[31487]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:28.105 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:27 smithi123 sudo[50631]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:28.105 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:27 smithi123 sudo[50631]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:28.105 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:27 smithi123 sudo[50631]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:28.106 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:27 smithi123 sudo[50631]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:28.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:27 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:28.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:27 smithi177 ceph-mon[36034]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:30:28.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:27 smithi177 ceph-mon[36034]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:27 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:30:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:27 smithi204 ceph-mon[36345]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:30:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:27 smithi204 ceph-mon[36345]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 -- 172.21.15.123:0/1855244165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4104310 msgr2=0x7f17f4104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 --2- 172.21.15.123:0/1855244165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4104310 0x7f17f4104730 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f17e40099c0 tx=0x7f17e4009cd0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 -- 172.21.15.123:0/1855244165 shutdown_connections 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 --2- 172.21.15.123:0/1855244165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105eb0 0x7f17f410c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 --2- 172.21.15.123:0/1855244165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4104310 0x7f17f4104730 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 --2- 172.21.15.123:0/1855244165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4105510 0x7f17f4105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.638+0000 7f17f3200700 1 -- 172.21.15.123:0/1855244165 >> 172.21.15.123:0/1855244165 conn(0x7f17f40ff8b0 msgr2=0x7f17f4101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.639+0000 7f17f3200700 1 -- 172.21.15.123:0/1855244165 shutdown_connections 2024-03-24T00:30:28.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.639+0000 7f17f3200700 1 -- 172.21.15.123:0/1855244165 wait complete. 2024-03-24T00:30:28.640 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:28 smithi123 sudo[50658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:28.640 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:28 smithi123 sudo[50658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:28.640 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:28 smithi123 sudo[50658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:28.640 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:30:28 smithi123 sudo[50658]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 Processor -- start 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 -- start start 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4104310 0x7f17f419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 0x7f17f41a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f410ff30 con 0x7f17f4105510 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f410fdb0 con 0x7f17f4105eb0 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.640+0000 7f17f3200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f41100b0 con 0x7f17f4104310 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50698/0 (socket says 172.21.15.123:50698) 2024-03-24T00:30:28.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f1400700 1 -- 172.21.15.123:0/3771073911 learned_addr learned my addr 172.21.15.123:0/3771073911 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f1e00700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4104310 0x7f17f419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 0x7f17f41a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 -- 172.21.15.123:0/3771073911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4104310 msgr2=0x7f17f419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4104310 0x7f17f419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 -- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 msgr2=0x7f17f419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f2800700 1 -- 172.21.15.123:0/3771073911 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f41a21a0 con 0x7f17f4105eb0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.641+0000 7f17f1400700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17f2800700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 0x7f17f41a1b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00efc0 tx=0x7f17ec00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec0092e0 con 0x7f17f4105eb0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17ec004bb0 con 0x7f17f4105eb0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec004280 con 0x7f17f4105eb0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f41a2430 con 0x7f17f4105eb0 2024-03-24T00:30:28.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.642+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f41a2920 con 0x7f17f4105eb0 2024-03-24T00:30:28.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.643+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f4068e50 con 0x7f17f4105eb0 2024-03-24T00:30:28.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.645+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f17ec01b030 con 0x7f17f4105eb0 2024-03-24T00:30:28.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.646+0000 7f17eaa00700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 0x7f17d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:28.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.646+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f17ec0cd120 con 0x7f17f4105eb0 2024-03-24T00:30:28.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.646+0000 7f17f1e00700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 0x7f17d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:28.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.647+0000 7f17f1e00700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 0x7f17d80a4d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b580 tx=0x7f17e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:28.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.650+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f17ec022020 con 0x7f17f4105eb0 2024-03-24T00:30:28.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:28.870+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f40625d0 con 0x7f17d80a2880 2024-03-24T00:30:28.905 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44673]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:28.906 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:28.906 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44673]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:28.906 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44673]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:29.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:28 smithi204 ceph-mon[36345]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:30:29.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:29.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:29.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:29.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:29.212 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44676]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:29.212 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44676]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:29.212 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44676]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:29.212 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:30:28 smithi177 sudo[44676]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:29.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:28 smithi177 ceph-mon[36034]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:30:29.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:29.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:29.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:29.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:29.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:28 smithi123 ceph-mon[31487]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:30:29.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:29.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:30:29.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:30:29.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:29.427 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45360]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:29.427 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45360]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:29.428 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45360]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:29.428 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45360]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:29.821 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45363]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:29.821 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45363]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:29.821 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45363]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:29.821 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:30:29 smithi204 sudo[45363]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50808]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50808]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50808]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:30.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:30.618+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f17ec012050 con 0x7f17f4105eb0 2024-03-24T00:30:30.802 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50827]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50827]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50827]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 sudo[50827]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 ceph-mon[31487]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 ceph-mon[31487]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:30.803 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:30.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:30 smithi177 ceph-mon[36034]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:30.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:30.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:30 smithi177 ceph-mon[36034]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:30.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:31.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:30 smithi204 sudo[45366]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:31.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:30 smithi204 sudo[45366]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:31.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:30 smithi204 sudo[45366]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:31.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:30 smithi204 sudo[45366]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 sudo[45369]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 sudo[45369]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 sudo[45369]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 sudo[45369]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:31.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:31.592 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44679]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:30:31.593 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44679]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:31.593 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44679]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:31.593 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44679]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:31 smithi123 ceph-mon[31487]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:31.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44682]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 sudo[44682]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:31.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:31.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:31.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:31.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:31.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:31 smithi177 ceph-mon[36034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:30:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:31 smithi204 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:30:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:32 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]: dispatch 2024-03-24T00:30:32.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:32 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]': finished 2024-03-24T00:30:32.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:32 smithi177 ceph-mon[36034]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:30:32.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:32 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:33.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:32 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]: dispatch 2024-03-24T00:30:33.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:32 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]': finished 2024-03-24T00:30:33.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:32 smithi204 ceph-mon[36345]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:30:33.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:32 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:33.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]: dispatch 2024-03-24T00:30:33.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4131788059' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8245aa9-2f8d-48bd-bad2-0b2e706438d2"}]': finished 2024-03-24T00:30:33.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:32 smithi123 ceph-mon[31487]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:30:33.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:32 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:34.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:33 smithi204 ceph-mon[36345]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 361 KiB/s wr, 24 op/s 2024-03-24T00:30:34.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:33 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1797238261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:34.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:33 smithi123 ceph-mon[31487]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 361 KiB/s wr, 24 op/s 2024-03-24T00:30:34.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:33 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1797238261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:34.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:33 smithi177 ceph-mon[36034]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 361 KiB/s wr, 24 op/s 2024-03-24T00:30:34.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:33 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1797238261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:30:36.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:36 smithi204 ceph-mon[36345]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 259 KiB/s wr, 17 op/s 2024-03-24T00:30:36.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:36 smithi123 ceph-mon[31487]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 259 KiB/s wr, 17 op/s 2024-03-24T00:30:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:36 smithi177 ceph-mon[36034]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 259 KiB/s wr, 17 op/s 2024-03-24T00:30:38.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:38 smithi204 ceph-mon[36345]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 202 KiB/s wr, 13 op/s 2024-03-24T00:30:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:38 smithi123 ceph-mon[31487]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 202 KiB/s wr, 13 op/s 2024-03-24T00:30:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:38 smithi177 ceph-mon[36034]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 202 KiB/s wr, 13 op/s 2024-03-24T00:30:40.281 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:40 smithi123 ceph-mon[31487]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:40 smithi204 ceph-mon[36345]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:40 smithi177 ceph-mon[36034]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:42 smithi204 ceph-mon[36345]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:30:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:42.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:42 smithi123 ceph-mon[31487]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:42.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:30:42.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:42 smithi177 ceph-mon[36034]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T00:30:42.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:30:42.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:43.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:43 smithi204 ceph-mon[36345]: Deploying daemon osd.3 on smithi123 2024-03-24T00:30:43.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:43 smithi123 ceph-mon[31487]: Deploying daemon osd.3 on smithi123 2024-03-24T00:30:43.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:43 smithi177 ceph-mon[36034]: Deploying daemon osd.3 on smithi123 2024-03-24T00:30:44.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:44 smithi123 ceph-mon[31487]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:44.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:44 smithi204 ceph-mon[36345]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:44.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:44 smithi177 ceph-mon[36034]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:46 smithi204 ceph-mon[36345]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:46.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:46 smithi123 ceph-mon[31487]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:46 smithi177 ceph-mon[36034]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:48.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:48 smithi204 ceph-mon[36345]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:48.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:48 smithi123 ceph-mon[31487]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:48 smithi177 ceph-mon[36034]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:50.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:50 smithi204 ceph-mon[36345]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:50.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:50 smithi123 ceph-mon[31487]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:50 smithi177 ceph-mon[36034]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:52.255 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:52 smithi123 ceph-mon[31487]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:52.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:52 smithi177 ceph-mon[36034]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:52 smithi204 ceph-mon[36345]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:54.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:54 smithi123 ceph-mon[31487]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:30:54.373 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:54 smithi204 ceph-mon[36345]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:30:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:54.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:54.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:54.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:54 smithi177 ceph-mon[36034]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:30:54.507 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 3 on host 'smithi123' 2024-03-24T00:30:54.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.505+0000 7f17eaa00700 1 -- 172.21.15.123:0/3771073911 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f17f40625d0 con 0x7f17d80a2880 2024-03-24T00:30:54.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.508+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 msgr2=0x7f17d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:54.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.508+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 0x7f17d80a4d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b580 tx=0x7f17e4005fb0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 msgr2=0x7f17f41a1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:54.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 0x7f17f41a1b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00efc0 tx=0x7f17ec00c3a0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 shutdown_connections 2024-03-24T00:30:54.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f17d80a2880 0x7f17d80a4d40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17f4105eb0 0x7f17f41a1b00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f4104310 0x7f17f419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 --2- 172.21.15.123:0/3771073911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f4105510 0x7f17f419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:54.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 >> 172.21.15.123:0/3771073911 conn(0x7f17f40ff8b0 msgr2=0x7f17f4108810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:54.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 shutdown_connections 2024-03-24T00:30:54.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:30:54.509+0000 7f17f3200700 1 -- 172.21.15.123:0/3771073911 wait complete. 2024-03-24T00:30:55.313 DEBUG:teuthology.orchestra.run.smithi123:osd.3> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T00:30:55.318 INFO:tasks.cephadm:Deploying osd.4 on smithi177 with /dev/vg_nvme/lv_3... 2024-03-24T00:30:55.319 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:30:55.523 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:30:55.523 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:30:55.523 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:30:55.524 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:55.524 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.524 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.524 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:55.524 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:55 smithi123 ceph-mon[31487]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:55.564 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:55.685 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:55 smithi177 ceph-mon[36034]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:55.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:30:55.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:55 smithi204 ceph-mon[36345]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:56 smithi177 ceph-mon[36034]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' 2024-03-24T00:30:56.816 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.817 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-mon[31487]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' 2024-03-24T00:30:56.817 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:30:56.387+0000 7f5b2f600700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T00:30:56.817 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:30:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:30:56.462+0000 7f5b24a00700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:56 smithi204 ceph-mon[36345]: from='osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693]' entity='osd.3' 2024-03-24T00:30:57.340 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:30:57.340 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:30:57.340 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-24T00:30:57.340 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-24T00:30:57.341 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110143 s, 952 MB/s 2024-03-24T00:30:57.341 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-24T00:30:57.497 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:30:57.497 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:30:57.497 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] boot 2024-03-24T00:30:57.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:30:57.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:57 smithi177 ceph-mon[36034]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] boot 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:57 smithi204 ceph-mon[36345]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: osd.3 [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] boot 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:30:57.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:57 smithi123 ceph-mon[31487]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:30:58.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-03-24T00:30:58.279 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:30:58.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:58 smithi177 ceph-mon[36034]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:30:59.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:58 smithi123 ceph-mon[31487]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:30:59.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:58 smithi204 ceph-mon[36345]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:30:59.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 -- 172.21.15.177:0/612361771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1010f0 msgr2=0x7f09fc101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 --2- 172.21.15.177:0/612361771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1010f0 0x7f09fc101510 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f09f80099c0 tx=0x7f09f8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 -- 172.21.15.177:0/612361771 shutdown_connections 2024-03-24T00:30:59.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 --2- 172.21.15.177:0/612361771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 --2- 172.21.15.177:0/612361771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1010f0 0x7f09fc101510 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 --2- 172.21.15.177:0/612361771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc1022f0 0x7f09fc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 -- 172.21.15.177:0/612361771 >> 172.21.15.177:0/612361771 conn(0x7f09fc0fc690 msgr2=0x7f09fc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 -- 172.21.15.177:0/612361771 shutdown_connections 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.791+0000 7f0a03407700 1 -- 172.21.15.177:0/612361771 wait complete. 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 Processor -- start 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 -- start start 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc1010f0 0x7f09fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc10de50 con 0x7f09fc1010f0 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc10dcd0 con 0x7f09fc1022f0 2024-03-24T00:30:59.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a03407700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc10dfd0 con 0x7f09fc102c90 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:60528/0 (socket says 172.21.15.177:60528) 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 -- 172.21.15.177:0/3742514732 learned_addr learned my addr 172.21.15.177:0/3742514732 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a019a4700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 -- 172.21.15.177:0/3742514732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 msgr2=0x7f09fc19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 -- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc1010f0 msgr2=0x7f09fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc1010f0 0x7f09fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.792+0000 7f0a009a2700 1 -- 172.21.15.177:0/3742514732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc1a0070 con 0x7f09fc1022f0 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f0a019a4700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:59.797 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f0a009a2700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00b530 tx=0x7f09ec00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:59.798 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec015020 con 0x7f09fc1022f0 2024-03-24T00:30:59.798 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09ec011d10 con 0x7f09fc1022f0 2024-03-24T00:30:59.798 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc109fb0 con 0x7f09fc1022f0 2024-03-24T00:30:59.798 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00f2e0 con 0x7f09fc1022f0 2024-03-24T00:30:59.798 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.793+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc10a570 con 0x7f09fc1022f0 2024-03-24T00:30:59.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.794+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc068e50 con 0x7f09fc1022f0 2024-03-24T00:30:59.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.795+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f09ec018020 con 0x7f09fc1022f0 2024-03-24T00:30:59.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.796+0000 7f09f27fc700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.796+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f09ec0cd060 con 0x7f09fc1022f0 2024-03-24T00:30:59.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.796+0000 7f0a011a3700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.800 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.797+0000 7f0a011a3700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 0x7f09e80a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098e0 tx=0x7f09f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:59.801 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:30:59.800+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09ec01f020 con 0x7f09fc1022f0 2024-03-24T00:31:00.010 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:00.009+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f09fc0625d0 con 0x7f09e80a2890 2024-03-24T00:31:00.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:31:00.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:00.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:00.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:30:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:31:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:00.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:00.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:30:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:31:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:00.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:30:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:01.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi123 2024-03-24T00:31:01.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi123 to 8545M 2024-03-24T00:31:01.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:31:01.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:01.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:01.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:01.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:00 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi123 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi123 to 8545M 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:01.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:01.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:00 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:01.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi123 2024-03-24T00:31:01.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi123 to 8545M 2024-03-24T00:31:01.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:31:01.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:01.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:01.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:01.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:00 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:02.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:01 smithi177 ceph-mon[36034]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:31:02.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:01 smithi177 ceph-mon[36034]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:02.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:01 smithi204 ceph-mon[36345]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:31:02.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:01 smithi204 ceph-mon[36345]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:01 smithi123 ceph-mon[31487]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:31:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:01 smithi123 ceph-mon[31487]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:03.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:02 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/1684411581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:02 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:02 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]': finished 2024-03-24T00:31:03.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:02 smithi177 ceph-mon[36034]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:31:03.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:03.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:02 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/1684411581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:02 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:02 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]': finished 2024-03-24T00:31:03.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:02 smithi204 ceph-mon[36345]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:31:03.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:02 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/1684411581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:02 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]: dispatch 2024-03-24T00:31:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:02 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26f71028-4c8f-4813-ba49-216b3919c913"}]': finished 2024-03-24T00:31:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:02 smithi123 ceph-mon[31487]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:31:03.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:04.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:03 smithi177 ceph-mon[36034]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-24T00:31:04.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:03 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/848572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:04.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:03 smithi204 ceph-mon[36345]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-24T00:31:04.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:03 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/848572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:04.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:03 smithi123 ceph-mon[31487]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-24T00:31:04.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:03 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/848572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:06.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:06 smithi204 ceph-mon[36345]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:06 smithi123 ceph-mon[31487]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:06 smithi177 ceph-mon[36034]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:08.097 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:08 smithi177 ceph-mon[36034]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T00:31:08.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:08 smithi204 ceph-mon[36345]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T00:31:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:08 smithi123 ceph-mon[31487]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T00:31:10.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:10 smithi177 ceph-mon[36034]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T00:31:10.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:31:10.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:10 smithi204 ceph-mon[36345]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T00:31:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:31:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:10.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:10 smithi123 ceph-mon[31487]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T00:31:10.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:31:10.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:11.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:11 smithi177 ceph-mon[36034]: Deploying daemon osd.4 on smithi177 2024-03-24T00:31:11.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:11 smithi204 ceph-mon[36345]: Deploying daemon osd.4 on smithi177 2024-03-24T00:31:11.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:11 smithi123 ceph-mon[31487]: Deploying daemon osd.4 on smithi177 2024-03-24T00:31:12.253 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:12 smithi177 ceph-mon[36034]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:31:12.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:12 smithi204 ceph-mon[36345]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:31:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:12 smithi123 ceph-mon[31487]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:31:14.207 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:14 smithi177 ceph-mon[36034]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:14 smithi204 ceph-mon[36345]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:14.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:14 smithi123 ceph-mon[31487]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:16.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:16 smithi177 ceph-mon[36034]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:16.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:16.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:16.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:16 smithi123 ceph-mon[31487]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:16.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:16 smithi204 ceph-mon[36345]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:18.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:18 smithi177 ceph-mon[36034]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:18.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:18 smithi123 ceph-mon[31487]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:18 smithi204 ceph-mon[36345]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:18.750 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 4 on host 'smithi177' 2024-03-24T00:31:18.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.749+0000 7f09f27fc700 1 -- 172.21.15.177:0/3742514732 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f09fc0625d0 con 0x7f09e80a2890 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 msgr2=0x7f09e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 0x7f09e80a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098e0 tx=0x7f09f8000bc0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 msgr2=0x7f09fc198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00b530 tx=0x7f09ec00b8f0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 shutdown_connections 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09e80a2890 0x7f09e80a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09fc102c90 0x7f09fc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc1010f0 0x7f09fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 --2- 172.21.15.177:0/3742514732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09fc1022f0 0x7f09fc198950 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 >> 172.21.15.177:0/3742514732 conn(0x7f09fc0fc690 msgr2=0x7f09fc1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 shutdown_connections 2024-03-24T00:31:18.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:31:18.752+0000 7f0a03407700 1 -- 172.21.15.177:0/3742514732 wait complete. 2024-03-24T00:31:19.355 DEBUG:teuthology.orchestra.run.smithi177:osd.4> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T00:31:19.358 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-03-24T00:31:19.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:31:19.609 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:31:20.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.256 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:20.257 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:21.010 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-mon[36034]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:31:21.011 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-mon[36034]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.011 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-mon[36034]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:31:21.011 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:21.011 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-mon[36034]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.011 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:31:20.751+0000 7f8bac7b2700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T00:31:21.011 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:31:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:31:20.759+0000 7f8ba85a5700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:31:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:20 smithi204 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:31:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:20 smithi204 ceph-mon[36345]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:20 smithi204 ceph-mon[36345]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:31:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:20 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:20 smithi204 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:20 smithi123 ceph-mon[31487]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:31:21.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:20 smithi123 ceph-mon[31487]: from='osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:20 smithi123 ceph-mon[31487]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:31:21.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:20 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:21.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:20 smithi123 ceph-mon[31487]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129234 s, 811 MB/s 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-24T00:31:21.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:21 smithi204 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:31:21.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:21 smithi204 ceph-mon[36345]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:31:21.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:21.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:21.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:21 smithi204 ceph-mon[36345]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:22.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:21 smithi123 ceph-mon[31487]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:31:22.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:21 smithi123 ceph-mon[31487]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:31:22.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:22.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:22.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:21 smithi123 ceph-mon[31487]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:22.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-03-24T00:31:22.146 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:21 smithi177 ceph-mon[36034]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:31:22.146 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:21 smithi177 ceph-mon[36034]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:31:22.146 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:22.146 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:22.147 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:21 smithi177 ceph-mon[36034]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:31:22.394 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:31:22.817 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:31:22.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:31:22.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:22.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] boot 2024-03-24T00:31:22.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:31:22.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:23.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:31:23.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:31:23.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:23.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] boot 2024-03-24T00:31:23.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:31:23.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:23.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:31:23.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:31:23.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:23.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: osd.4 [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] boot 2024-03-24T00:31:23.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:31:23.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:31:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:31:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi177 2024-03-24T00:31:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:23.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:23.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:23.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 -- 172.21.15.204:0/2040109818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8101a60 msgr2=0x7fc4a8101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/2040109818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8101a60 0x7fc4a8101ec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009990 tx=0x7fc49c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 -- 172.21.15.204:0/2040109818 shutdown_connections 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/2040109818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a8102400 0x7fc4a8108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/2040109818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 0x7fc4a8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/2040109818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8101a60 0x7fc4a8101ec0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 -- 172.21.15.204:0/2040109818 >> 172.21.15.204:0/2040109818 conn(0x7fc4a80fbde0 msgr2=0x7fc4a80fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 -- 172.21.15.204:0/2040109818 shutdown_connections 2024-03-24T00:31:23.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 -- 172.21.15.204:0/2040109818 wait complete. 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.864+0000 7fc4ae07e700 1 Processor -- start 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 -- start start 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 0x7fc4a8197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a8101a60 0x7fc4a8197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a810cd40 con 0x7fc4a8101a60 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a810cbc0 con 0x7fc4a8100860 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4ae07e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a810cec0 con 0x7fc4a8102400 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:35888/0 (socket says 172.21.15.204:35888) 2024-03-24T00:31:23.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a77fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 0x7fc4a8197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 -- 172.21.15.204:0/201654796 learned_addr learned my addr 172.21.15.204:0/201654796 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 -- 172.21.15.204:0/201654796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 msgr2=0x7fc4a8197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 0x7fc4a8197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 -- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a8101a60 msgr2=0x7fc4a8197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a8101a60 0x7fc4a8197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 -- 172.21.15.204:0/201654796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a819ef90 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a7fff700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b9a0 tx=0x7fc4a000bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.865+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0019070 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.866+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a819f280 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.866+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4a000ce10 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.866+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00073d0 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.866+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a819f840 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.866+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a8068e50 con 0x7fc4a8102400 2024-03-24T00:31:23.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.868+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc4a001c070 con 0x7fc4a8102400 2024-03-24T00:31:23.872 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.869+0000 7fc4a4ff9700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 0x7fc4900a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.872 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.869+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fc4a00cd2b0 con 0x7fc4a8102400 2024-03-24T00:31:23.872 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.869+0000 7fc4a77fe700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 0x7fc4900a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.872 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.870+0000 7fc4a77fe700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 0x7fc4900a4dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81018c0 tx=0x7fc498009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:23.872 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:23.873+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc4a009b320 con 0x7fc4a8102400 2024-03-24T00:31:24.079 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:24.079+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4a80625d0 con 0x7fc4900a2910 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi177 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:24.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:24.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:24.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:24.108 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:31:24.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi177 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:24.996 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:24 smithi204 ceph-mon[36345]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:24.996 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:24 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:24.996 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:24 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:24.996 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:24 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:25.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:24 smithi177 ceph-mon[36034]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:25.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:24 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:25.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:24 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:25.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:24 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:25.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:24 smithi123 ceph-mon[31487]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:25.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:24 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:25.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:24 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:25.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:24 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:26.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:25 smithi177 ceph-mon[36034]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:26.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:25 smithi204 ceph-mon[36345]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:26.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:25 smithi123 ceph-mon[31487]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:27.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:26 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/1267633020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:26 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:26 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]': finished 2024-03-24T00:31:27.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:26 smithi177 ceph-mon[36034]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:31:27.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:27.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:26 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/1267633020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:26 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:26 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]': finished 2024-03-24T00:31:27.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:26 smithi204 ceph-mon[36345]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:31:27.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:27.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:26 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/1267633020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:26 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]: dispatch 2024-03-24T00:31:27.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:26 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34571dde-a4d1-4a14-adbc-55097581ca68"}]': finished 2024-03-24T00:31:27.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:26 smithi123 ceph-mon[31487]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:31:27.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:27 smithi204 ceph-mon[36345]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:27 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/3466940444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:28.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:27 smithi123 ceph-mon[31487]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:28.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:27 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/3466940444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:27 smithi177 ceph-mon[36034]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:27 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/3466940444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:30 smithi204 ceph-mon[36345]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:30 smithi123 ceph-mon[31487]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:30 smithi177 ceph-mon[36034]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:32.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:32 smithi204 ceph-mon[36345]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:32.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:32 smithi123 ceph-mon[31487]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:32.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:32 smithi177 ceph-mon[36034]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:34.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:34 smithi204 ceph-mon[36345]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:34.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:34 smithi123 ceph-mon[31487]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:34 smithi177 ceph-mon[36034]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:35.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:31:35.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:35 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:35.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:35 smithi204 ceph-mon[36345]: Deploying daemon osd.5 on smithi204 2024-03-24T00:31:35.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:31:35.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:35 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:35.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:35 smithi123 ceph-mon[31487]: Deploying daemon osd.5 on smithi204 2024-03-24T00:31:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:31:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:35 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:35 smithi177 ceph-mon[36034]: Deploying daemon osd.5 on smithi204 2024-03-24T00:31:36.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:36 smithi204 ceph-mon[36345]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:36.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:36 smithi123 ceph-mon[31487]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:36 smithi177 ceph-mon[36034]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:38.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:38 smithi204 ceph-mon[36345]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:38 smithi123 ceph-mon[31487]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:38 smithi177 ceph-mon[36034]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:40.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:40 smithi204 ceph-mon[36345]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:40.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:40 smithi123 ceph-mon[31487]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:40 smithi177 ceph-mon[36034]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:40.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:42.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:42 smithi204 ceph-mon[36345]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:42.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:42 smithi123 ceph-mon[31487]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:42.417 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.416+0000 7fc4a4ff9700 1 -- 172.21.15.204:0/201654796 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc4a80625d0 con 0x7fc4900a2910 2024-03-24T00:31:42.417 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.418+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 msgr2=0x7fc4900a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 0x7fc4900a4dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81018c0 tx=0x7fc498009260 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 msgr2=0x7fc4a819e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b9a0 tx=0x7fc4a000bcb0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 shutdown_connections 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4900a2910 0x7fc4900a4dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a8102400 0x7fc4a819e8f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a8100860 0x7fc4a8197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 --2- 172.21.15.204:0/201654796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a8101a60 0x7fc4a8197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 >> 172.21.15.204:0/201654796 conn(0x7fc4a80fbde0 msgr2=0x7fc4a8104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.419 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 shutdown_connections 2024-03-24T00:31:42.420 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:31:42.419+0000 7fc4ae07e700 1 -- 172.21.15.204:0/201654796 wait complete. 2024-03-24T00:31:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:42 smithi177 ceph-mon[36034]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:43.009 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T00:31:43.012 INFO:tasks.cephadm:Deploying osd.6 on smithi123 with /dev/vg_nvme/lv_2... 2024-03-24T00:31:43.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:31:43.321 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:31:43.187+0000 7f4923adb7c0 -1 osd.5 0 log_to_monitors true 2024-03-24T00:31:43.462 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:31:43.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:43 smithi204 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:43.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:43 smithi177 ceph-mon[36034]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:43.765 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.765 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.765 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:43.765 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.765 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:43.766 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:43.766 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:43 smithi123 ceph-mon[31487]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:31:45.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:45 smithi123 ceph-mon[31487]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:31:45.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:45 smithi123 ceph-mon[31487]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:45 smithi123 ceph-mon[31487]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:31:45.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:45.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:45 smithi123 ceph-mon[31487]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:45 smithi177 ceph-mon[36034]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:31:45.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:45 smithi177 ceph-mon[36034]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:45 smithi177 ceph-mon[36034]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:31:45.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:45.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:45 smithi177 ceph-mon[36034]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:31:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-mon[36345]: from='osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-mon[36345]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:31:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:31:45.572 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:31:45.079+0000 7f4919140700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T00:31:45.572 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:31:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:31:45.086+0000 7f4914f33700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] boot 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:31:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] boot 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:31:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0359081 s, 292 MB/s 2024-03-24T00:31:46.365 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-24T00:31:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:31:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:31:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:31:46.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:46.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: osd.5 [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] boot 2024-03-24T00:31:46.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:31:46.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:31:47.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:47 smithi123 ceph-mon[31487]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:31:47.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:31:47.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:31:47.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:47 smithi177 ceph-mon[36034]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:31:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:31:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:31:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:31:47.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:47 smithi204 ceph-mon[36345]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:31:47.585 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:31:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:48 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi204 2024-03-24T00:31:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:48 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi204 to 8540M 2024-03-24T00:31:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:48 smithi123 ceph-mon[31487]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:48 smithi123 ceph-mon[31487]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:31:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:48 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi204 2024-03-24T00:31:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:48 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi204 to 8540M 2024-03-24T00:31:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:48 smithi177 ceph-mon[36034]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:48 smithi177 ceph-mon[36034]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:31:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:48 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi204 2024-03-24T00:31:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:48 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi204 to 8540M 2024-03-24T00:31:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:48 smithi204 ceph-mon[36345]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:48 smithi204 ceph-mon[36345]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 -- 172.21.15.123:0/1304021180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce5410a3b0 msgr2=0x7fce54110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 --2- 172.21.15.123:0/1304021180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce5410a3b0 0x7fce54110c60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009fc0 tx=0x7fce4c00b330 comp rx=0 tx=0).stop 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 -- 172.21.15.123:0/1304021180 shutdown_connections 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 --2- 172.21.15.123:0/1304021180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce5410a3b0 0x7fce54110c60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 --2- 172.21.15.123:0/1304021180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce54108810 0x7fce54108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 --2- 172.21.15.123:0/1304021180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54109a10 0x7fce54109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 -- 172.21.15.123:0/1304021180 >> 172.21.15.123:0/1304021180 conn(0x7fce540777e0 msgr2=0x7fce54079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 -- 172.21.15.123:0/1304021180 shutdown_connections 2024-03-24T00:31:49.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.778+0000 7fce59a00700 1 -- 172.21.15.123:0/1304021180 wait complete. 2024-03-24T00:31:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.779+0000 7fce59a00700 1 Processor -- start 2024-03-24T00:31:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 -- start start 2024-03-24T00:31:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 0x7fce5419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce5410a3b0 0x7fce541a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce54114430 con 0x7fce54108810 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce541142b0 con 0x7fce54109a10 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce59a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce541145b0 con 0x7fce5410a3b0 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce52000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce52000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46150/0 (socket says 172.21.15.123:46150) 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce52000700 1 -- 172.21.15.123:0/1440808376 learned_addr learned my addr 172.21.15.123:0/1440808376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce52a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 0x7fce5419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.780+0000 7fce53400700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce5410a3b0 0x7fce541a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 -- 172.21.15.123:0/1440808376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce5410a3b0 msgr2=0x7fce541a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce5410a3b0 0x7fce541a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 -- 172.21.15.123:0/1440808376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 msgr2=0x7fce5419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 -- 172.21.15.123:0/1440808376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce541a65e0 con 0x7fce54108810 2024-03-24T00:31:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52000700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce52a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 0x7fce5419e980 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e9f0 tx=0x7fce4400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4400cb70 con 0x7fce54108810 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.781+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce541a68d0 con 0x7fce54108810 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.782+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce4400ee90 con 0x7fce54108810 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.782+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44018680 con 0x7fce54108810 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.782+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce541a6e60 con 0x7fce54108810 2024-03-24T00:31:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.783+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce54068e50 con 0x7fce54108810 2024-03-24T00:31:49.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.785+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fce4401f020 con 0x7fce54108810 2024-03-24T00:31:49.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.786+0000 7fce4b400700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 0x7fce380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.786+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7fce440cdaa0 con 0x7fce54108810 2024-03-24T00:31:49.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.787+0000 7fce52000700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 0x7fce380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.787+0000 7fce52000700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 0x7fce380a4dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fce40005950 tx=0x7fce40009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:49.791+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce44053740 con 0x7fce54108810 2024-03-24T00:31:50.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:31:50.023+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fce54062690 con 0x7fce380a2910 2024-03-24T00:31:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:50 smithi123 ceph-mon[31487]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:50 smithi123 ceph-mon[31487]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:31:50.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:50.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:50.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:50 smithi177 ceph-mon[36034]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:50 smithi177 ceph-mon[36034]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:31:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:50 smithi204 ceph-mon[36345]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:50 smithi204 ceph-mon[36345]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:31:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:31:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:31:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:51.296 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:51 smithi123 ceph-mon[31487]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:51.296 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:51 smithi123 ceph-mon[31487]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:31:51.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:51 smithi177 ceph-mon[36034]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:51.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:51 smithi177 ceph-mon[36034]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:31:51.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:51 smithi204 ceph-mon[36345]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:31:51.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:51 smithi204 ceph-mon[36345]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:31:52.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:52 smithi177 ceph-mon[36034]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:52 smithi204 ceph-mon[36345]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:52.572 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:52 smithi123 ceph-mon[31487]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:31:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:54 smithi204 ceph-mon[36345]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]: dispatch 2024-03-24T00:31:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]': finished 2024-03-24T00:31:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:54 smithi204 ceph-mon[36345]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:31:54.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:31:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:54 smithi177 ceph-mon[36034]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]: dispatch 2024-03-24T00:31:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]': finished 2024-03-24T00:31:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:54 smithi177 ceph-mon[36034]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:31:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:31:54.587 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:54 smithi123 ceph-mon[31487]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:54.587 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]: dispatch 2024-03-24T00:31:54.587 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3289416575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a965499-6acb-447a-be25-fe19655bf999"}]': finished 2024-03-24T00:31:54.587 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:54 smithi123 ceph-mon[31487]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:31:54.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:31:55.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/707531215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:55.568 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/707531215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:55.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/707531215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:31:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:56 smithi177 ceph-mon[36034]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:56.465 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:56 smithi123 ceph-mon[31487]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:56 smithi204 ceph-mon[36345]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:31:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:31:58 smithi177 ceph-mon[36034]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T00:31:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:31:58 smithi204 ceph-mon[36345]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T00:31:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:31:58 smithi123 ceph-mon[31487]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T00:32:00.447 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:00 smithi123 ceph-mon[31487]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T00:32:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:00 smithi177 ceph-mon[36034]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T00:32:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:00 smithi204 ceph-mon[36345]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T00:32:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:02 smithi177 ceph-mon[36034]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:32:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:02 smithi204 ceph-mon[36345]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:32:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:02 smithi123 ceph-mon[31487]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:32:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:04 smithi177 ceph-mon[36034]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:32:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:04.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:04 smithi123 ceph-mon[31487]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:04.539 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:32:04.539 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:04 smithi204 ceph-mon[36345]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:32:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:05.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:05 smithi177 ceph-mon[36034]: Deploying daemon osd.6 on smithi123 2024-03-24T00:32:05.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:05 smithi204 ceph-mon[36345]: Deploying daemon osd.6 on smithi123 2024-03-24T00:32:05.572 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:05 smithi123 ceph-mon[31487]: Deploying daemon osd.6 on smithi123 2024-03-24T00:32:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:06 smithi123 ceph-mon[31487]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:06 smithi177 ceph-mon[36034]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:06 smithi204 ceph-mon[36345]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:08.443 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:08 smithi123 ceph-mon[31487]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:08.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:08 smithi177 ceph-mon[36034]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:08 smithi204 ceph-mon[36345]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:10 smithi204 ceph-mon[36345]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:10 smithi123 ceph-mon[31487]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:10 smithi177 ceph-mon[36034]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:12 smithi123 ceph-mon[31487]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:12.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:12 smithi204 ceph-mon[36345]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:12.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:12 smithi177 ceph-mon[36034]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:14 smithi204 ceph-mon[36345]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:14 smithi123 ceph-mon[31487]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:14 smithi177 ceph-mon[36034]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:16.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:16.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:16 smithi177 ceph-mon[36034]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:32:16.503 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:16.503 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.504 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.504 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:16.504 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:16.504 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.504 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:16 smithi123 ceph-mon[31487]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:16 smithi204 ceph-mon[36345]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:32:16.601 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 6 on host 'smithi123' 2024-03-24T00:32:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.599+0000 7fce4b400700 1 -- 172.21.15.123:0/1440808376 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fce54062690 con 0x7fce380a2910 2024-03-24T00:32:16.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.602+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 msgr2=0x7fce380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:16.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.602+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 0x7fce380a4dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fce40005950 tx=0x7fce40009310 comp rx=0 tx=0).stop 2024-03-24T00:32:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.602+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 msgr2=0x7fce5419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.602+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 0x7fce5419e980 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e9f0 tx=0x7fce4400ed00 comp rx=0 tx=0).stop 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 shutdown_connections 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fce380a2910 0x7fce380a4dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce5410a3b0 0x7fce541a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce54108810 0x7fce5419e980 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 --2- 172.21.15.123:0/1440808376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce54109a10 0x7fce5419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 >> 172.21.15.123:0/1440808376 conn(0x7fce540777e0 msgr2=0x7fce5410ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 shutdown_connections 2024-03-24T00:32:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:32:16.603+0000 7fce59a00700 1 -- 172.21.15.123:0/1440808376 wait complete. 2024-03-24T00:32:17.256 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.257 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:17.367 DEBUG:teuthology.orchestra.run.smithi123:osd.6> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T00:32:17.371 INFO:tasks.cephadm:Deploying osd.7 on smithi177 with /dev/vg_nvme/lv_2... 2024-03-24T00:32:17.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:32:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:32:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:32:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:32:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:17.619 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:32:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:32:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:32:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:32:17.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:17.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:17.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:18.406 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:18 smithi177 ceph-mon[36034]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:18.407 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:18 smithi177 ceph-mon[36034]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:32:18.407 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:18 smithi177 ceph-mon[36034]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:32:18.407 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:18 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:18.407 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:18 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:18 smithi204 ceph-mon[36345]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:18 smithi204 ceph-mon[36345]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:32:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:18 smithi204 ceph-mon[36345]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:32:18.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:18 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:18.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:18 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:18 smithi123 ceph-mon[31487]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:18 smithi123 ceph-mon[31487]: from='osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:32:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:18 smithi123 ceph-mon[31487]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:32:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:18 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:18 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:19.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:19 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:32:19.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:19 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:32:19.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:19 smithi123 ceph-mon[31487]: osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] boot 2024-03-24T00:32:19.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:19 smithi123 ceph-mon[31487]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:32:19.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:19.520 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:32:19.520 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:32:19.520 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-24T00:32:19.520 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-24T00:32:19.521 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01093 s, 959 MB/s 2024-03-24T00:32:19.521 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-24T00:32:19.521 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:19 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:32:19.521 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:19 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:32:19.521 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:19 smithi177 ceph-mon[36034]: osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] boot 2024-03-24T00:32:19.521 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:19 smithi177 ceph-mon[36034]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:32:19.521 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:19 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:32:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:19 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:32:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:19 smithi204 ceph-mon[36345]: osd.6 [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] boot 2024-03-24T00:32:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:19 smithi204 ceph-mon[36345]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:32:19.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:32:20.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-03-24T00:32:20.359 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:32:20.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:20 smithi177 ceph-mon[36034]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:20.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:20 smithi177 ceph-mon[36034]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:32:20.463 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:20 smithi123 ceph-mon[31487]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:20.463 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:20 smithi123 ceph-mon[31487]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:32:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:20 smithi204 ceph-mon[36345]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:32:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:20 smithi204 ceph-mon[36345]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:32:21.442 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:21 smithi177 ceph-mon[36034]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:21.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:21 smithi204 ceph-mon[36345]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:21.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:21 smithi123 ceph-mon[31487]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 -- 172.21.15.177:0/3507938833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1838103d60 msgr2=0x7f183810a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 --2- 172.21.15.177:0/3507938833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1838103d60 0x7f183810a610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f183000b0d0 tx=0x7f183000b3e0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 -- 172.21.15.177:0/3507938833 shutdown_connections 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 --2- 172.21.15.177:0/3507938833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1838103d60 0x7f183810a610 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 --2- 172.21.15.177:0/3507938833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381021c0 0x7f18381025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 --2- 172.21.15.177:0/3507938833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18381033c0 0x7f1838103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 -- 172.21.15.177:0/3507938833 >> 172.21.15.177:0/3507938833 conn(0x7f18380fd7a0 msgr2=0x7f18380ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 -- 172.21.15.177:0/3507938833 shutdown_connections 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.916+0000 7f183ec5e700 1 -- 172.21.15.177:0/3507938833 wait complete. 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 Processor -- start 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 -- start start 2024-03-24T00:32:21.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381033c0 0x7f1838198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1838103d60 0x7f183819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183810de30 con 0x7f18381033c0 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183810dcb0 con 0x7f18381021c0 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183ec5e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183810dfb0 con 0x7f1838103d60 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:42278/0 (socket says 172.21.15.177:42278) 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 -- 172.21.15.177:0/224608064 learned_addr learned my addr 172.21.15.177:0/224608064 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183d1fb700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1838103d60 0x7f183819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f182ffff700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381033c0 0x7f1838198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 -- 172.21.15.177:0/224608064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1838103d60 msgr2=0x7f183819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1838103d60 0x7f183819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 -- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381033c0 msgr2=0x7f1838198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381033c0 0x7f1838198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 -- 172.21.15.177:0/224608064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18381a0020 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f183c9fa700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f183400b500 tx=0x7f183400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.917+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834015020 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.918+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18381a0310 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.918+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1834011d10 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.918+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183400f2e0 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.918+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18381a0920 con 0x7f18381021c0 2024-03-24T00:32:21.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.919+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1838068e50 con 0x7f18381021c0 2024-03-24T00:32:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.920+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1834018020 con 0x7f18381021c0 2024-03-24T00:32:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.920+0000 7f182dffb700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 0x7f18200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.920+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f18340cd4e0 con 0x7f18381021c0 2024-03-24T00:32:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.921+0000 7f182ffff700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 0x7f18200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.921+0000 7f182ffff700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 0x7f18200a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1828005950 tx=0x7f18280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:21.925+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f183401f020 con 0x7f18381021c0 2024-03-24T00:32:22.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:22.129+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f18380625d0 con 0x7f18200a2890 2024-03-24T00:32:22.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:22 smithi123 ceph-mon[31487]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:22.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:22.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:22.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:22 smithi177 ceph-mon[36034]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:22 smithi204 ceph-mon[36345]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:24.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi123 2024-03-24T00:32:24.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi123 to 5697M 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:24.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi123 2024-03-24T00:32:24.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi123 to 5697M 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi123 2024-03-24T00:32:24.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi123 to 5697M 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:24.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:25.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:25 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/2310243404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:25 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:25 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]': finished 2024-03-24T00:32:25.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:25 smithi204 ceph-mon[36345]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:32:25.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:25.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:25 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/2310243404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:25 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:25 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]': finished 2024-03-24T00:32:25.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:25 smithi123 ceph-mon[31487]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:32:25.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:25.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:25 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/2310243404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:25 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]: dispatch 2024-03-24T00:32:25.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:25 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c47f489-d125-48a6-8013-9702d5bf03f6"}]': finished 2024-03-24T00:32:25.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:25 smithi177 ceph-mon[36034]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:32:25.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:26.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:26 smithi204 ceph-mon[36345]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:26.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:26 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/117810538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:26.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:26 smithi123 ceph-mon[31487]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:26.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:26 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/117810538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:26 smithi177 ceph-mon[36034]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:26 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/117810538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:28.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:28 smithi204 ceph-mon[36345]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:28.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:28 smithi123 ceph-mon[31487]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:28 smithi177 ceph-mon[36034]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:30.197 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:30 smithi177 ceph-mon[36034]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:30.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:30 smithi204 ceph-mon[36345]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:30.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:30 smithi123 ceph-mon[31487]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:32.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:32 smithi177 ceph-mon[36034]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:32.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:32 smithi204 ceph-mon[36345]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:32.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:32 smithi123 ceph-mon[31487]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:33.262 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:32:33.262 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:33.262 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:33 smithi177 ceph-mon[36034]: Deploying daemon osd.7 on smithi177 2024-03-24T00:32:33.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:32:33.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:33.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:33 smithi204 ceph-mon[36345]: Deploying daemon osd.7 on smithi177 2024-03-24T00:32:33.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:32:33.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:33.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:33 smithi123 ceph-mon[31487]: Deploying daemon osd.7 on smithi177 2024-03-24T00:32:34.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:34 smithi177 ceph-mon[36034]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:34.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:34 smithi204 ceph-mon[36345]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:34.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:34 smithi123 ceph-mon[31487]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:36.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:36 smithi177 ceph-mon[36034]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:36.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:36 smithi204 ceph-mon[36345]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:36.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:36 smithi123 ceph-mon[31487]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:38.173 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:38 smithi177 ceph-mon[36034]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:38.173 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:38 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:38.173 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:38 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:38.173 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:38 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:38.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:38 smithi204 ceph-mon[36345]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:38.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:38 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:38.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:38 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:38.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:38 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:38 smithi123 ceph-mon[31487]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:38 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:38 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:38 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:40.129 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:40 smithi177 ceph-mon[36034]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:40 smithi204 ceph-mon[36345]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:40 smithi123 ceph-mon[31487]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:41.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.157+0000 7f182dffb700 1 -- 172.21.15.177:0/224608064 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f18380625d0 con 0x7f18200a2890 2024-03-24T00:32:41.158 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 7 on host 'smithi177' 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 msgr2=0x7f18200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 0x7f18200a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1828005950 tx=0x7f18280058e0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 msgr2=0x7f18381983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f183400b500 tx=0x7f183400b810 comp rx=0 tx=0).stop 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 shutdown_connections 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f18200a2890 0x7f18200a4d50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1838103d60 0x7f183819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18381021c0 0x7f18381983c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 --2- 172.21.15.177:0/224608064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18381033c0 0x7f1838198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 >> 172.21.15.177:0/224608064 conn(0x7f18380fd7a0 msgr2=0x7f18381066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 shutdown_connections 2024-03-24T00:32:41.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:32:41.160+0000 7f183ec5e700 1 -- 172.21.15.177:0/224608064 wait complete. 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:41.764 DEBUG:teuthology.orchestra.run.smithi177:osd.7> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T00:32:41.766 INFO:tasks.cephadm:Deploying osd.8 on smithi204 with /dev/vg_nvme/lv_2... 2024-03-24T00:32:41.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:32:42.012 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:32:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:42 smithi177 ceph-mon[36034]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:42 smithi177 ceph-mon[36034]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:42.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:42 smithi177 ceph-mon[36034]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:42.467 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:42 smithi204 ceph-mon[36345]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:42.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:42 smithi204 ceph-mon[36345]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:42.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:42 smithi204 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:42.508 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:42 smithi123 ceph-mon[31487]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:42.508 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:42 smithi123 ceph-mon[31487]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:42.508 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:42 smithi123 ceph-mon[31487]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:32:43.279 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:32:43.276+0000 7f7ff348b700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.533 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:43 smithi123 ceph-mon[31487]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:32:43.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:32:43.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:43.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:43 smithi204 ceph-mon[36345]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:43.694 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:32:43.287+0000 7f7fef27e700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: from='osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:32:43.694 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:43 smithi177 ceph-mon[36034]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113005 s, 928 MB/s 2024-03-24T00:32:43.821 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-24T00:32:44.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:32:44.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:32:44.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:44.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] boot 2024-03-24T00:32:44.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:32:44.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:44 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:44.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] boot 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:32:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:44 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:44.693 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: osd.7 [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] boot 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:32:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:44 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:32:45.392 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:32:45.392 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:32:45.392 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi177 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:45.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:45 smithi204 ceph-mon[36345]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi177 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:45.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:45 smithi123 ceph-mon[31487]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:32:45.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:32:45.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:32:45.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi177 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:45.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:45 smithi177 ceph-mon[36034]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 -- 172.21.15.204:0/2433054330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 msgr2=0x7efea8101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 --2- 172.21.15.204:0/2433054330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8101ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe94009990 tx=0x7efe94009ca0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 -- 172.21.15.204:0/2433054330 shutdown_connections 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 --2- 172.21.15.204:0/2433054330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea8108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 --2- 172.21.15.204:0/2433054330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea8100840 0x7efea8100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 --2- 172.21.15.204:0/2433054330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8101ea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 -- 172.21.15.204:0/2433054330 >> 172.21.15.204:0/2433054330 conn(0x7efea80fbe00 msgr2=0x7efea80fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 -- 172.21.15.204:0/2433054330 shutdown_connections 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.224+0000 7efeaed60700 1 -- 172.21.15.204:0/2433054330 wait complete. 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 Processor -- start 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 -- start start 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea8100840 0x7efea8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea810deb0 con 0x7efea8100840 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea810dd30 con 0x7efea81023e0 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efeaed60700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea810e030 con 0x7efea8101a40 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:51154/0 (socket says 172.21.15.204:51154) 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 -- 172.21.15.204:0/3443685174 learned_addr learned my addr 172.21.15.204:0/3443685174 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 -- 172.21.15.204:0/3443685174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 msgr2=0x7efea819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.228 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efead2fd700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 -- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea8100840 msgr2=0x7efea8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea8100840 0x7efea8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efea7fff700 1 -- 172.21.15.204:0/3443685174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea81a01c0 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.225+0000 7efead2fd700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efea7fff700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe940047b0 tx=0x7efe94003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94027070 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe94022470 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea81a0450 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9400f6c0 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.226+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea81a0920 con 0x7efea8101a40 2024-03-24T00:32:46.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.227+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea8068e50 con 0x7efea8101a40 2024-03-24T00:32:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.228+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efe9402a070 con 0x7efea8101a40 2024-03-24T00:32:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.229+0000 7efea5ffb700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 0x7efe980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.229+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7efe940cf510 con 0x7efea8101a40 2024-03-24T00:32:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.229+0000 7efeacafc700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 0x7efe980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.230+0000 7efeacafc700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 0x7efe980a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efea81018a0 tx=0x7efe9c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:46.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.233+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe94031070 con 0x7efea8101a40 2024-03-24T00:32:46.439 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:32:46.439+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efea80625d0 con 0x7efe980a2890 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:47.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:47 smithi204 ceph-mon[36345]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:47.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:32:47.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:47.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:47.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:47.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:47.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:47 smithi123 ceph-mon[31487]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:47.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:32:47.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:32:47.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:32:47.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:32:47.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:47.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:47 smithi177 ceph-mon[36034]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:48 smithi204 ceph-mon[36345]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:32:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:48 smithi204 ceph-mon[36345]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:32:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:48 smithi123 ceph-mon[31487]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:32:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:48 smithi123 ceph-mon[31487]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:32:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:48 smithi177 ceph-mon[36034]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:32:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:48 smithi177 ceph-mon[36034]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/2468257554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]': finished 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:32:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:32:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:32:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:49.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/2468257554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]': finished 2024-03-24T00:32:49.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:32:49.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:32:49.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:32:49.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:49.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/2468257554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]: dispatch 2024-03-24T00:32:49.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7"}]': finished 2024-03-24T00:32:49.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:32:49.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:49 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:32:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:50 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/68968868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:50 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/68968868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:50 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/68968868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:32:51.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:51 smithi204 ceph-mon[36345]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:51 smithi123 ceph-mon[31487]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:51 smithi177 ceph-mon[36034]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:52 smithi123 ceph-mon[31487]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T00:32:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:52 smithi123 ceph-mon[31487]: Cluster is now healthy 2024-03-24T00:32:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:52 smithi177 ceph-mon[36034]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T00:32:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:52 smithi177 ceph-mon[36034]: Cluster is now healthy 2024-03-24T00:32:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:52 smithi204 ceph-mon[36345]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T00:32:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:52 smithi204 ceph-mon[36345]: Cluster is now healthy 2024-03-24T00:32:53.568 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:53 smithi204 ceph-mon[36345]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:53.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:53 smithi123 ceph-mon[31487]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:53 smithi177 ceph-mon[36034]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:56.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:56 smithi204 ceph-mon[36345]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:56.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:56 smithi123 ceph-mon[31487]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:56 smithi177 ceph-mon[36034]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:57.317 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:32:57.317 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:57.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:32:57.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:57.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:32:57.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:58.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:58 smithi204 ceph-mon[36345]: Deploying daemon osd.8 on smithi204 2024-03-24T00:32:58.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:32:58 smithi204 ceph-mon[36345]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:58.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:58 smithi123 ceph-mon[31487]: Deploying daemon osd.8 on smithi204 2024-03-24T00:32:58.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:32:58 smithi123 ceph-mon[31487]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:32:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:58 smithi177 ceph-mon[36034]: Deploying daemon osd.8 on smithi204 2024-03-24T00:32:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:32:58 smithi177 ceph-mon[36034]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:00.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:00 smithi204 ceph-mon[36345]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:00.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:00 smithi123 ceph-mon[31487]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:00 smithi177 ceph-mon[36034]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:02 smithi123 ceph-mon[31487]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:02.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:02 smithi177 ceph-mon[36034]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:02.473 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:02 smithi204 ceph-mon[36345]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:02.473 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:02.473 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:02.474 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:04.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:04 smithi204 ceph-mon[36345]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:04.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:04 smithi123 ceph-mon[31487]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:04 smithi177 ceph-mon[36034]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:05.273 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 8 on host 'smithi204' 2024-03-24T00:33:05.274 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.272+0000 7efea5ffb700 1 -- 172.21.15.204:0/3443685174 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efea80625d0 con 0x7efe980a2890 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.274+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 msgr2=0x7efe980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.274+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 0x7efe980a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efea81018a0 tx=0x7efe9c006d20 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.274+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 msgr2=0x7efea8198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.274+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe940047b0 tx=0x7efe94003680 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 shutdown_connections 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efe980a2890 0x7efe980a4d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efea81023e0 0x7efea819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea8100840 0x7efea8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 --2- 172.21.15.204:0/3443685174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efea8101a40 0x7efea8198aa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 >> 172.21.15.204:0/3443685174 conn(0x7efea80fbe00 msgr2=0x7efea8104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 shutdown_connections 2024-03-24T00:33:05.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:33:05.275+0000 7efeaed60700 1 -- 172.21.15.204:0/3443685174 wait complete. 2024-03-24T00:33:05.892 DEBUG:teuthology.orchestra.run.smithi204:osd.8> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T00:33:05.894 INFO:tasks.cephadm:Deploying osd.9 on smithi123 with /dev/vg_nvme/lv_1... 2024-03-24T00:33:05.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:33:06.340 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:33:06.546 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:06.546 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.546 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.547 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.547 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.547 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.547 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:33:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:07.549 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:33:07.549 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-mon[36345]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:33:07.549 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:07.550 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-mon[36345]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:07.550 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:07.550 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:33:07.253+0000 7f7ab5d15700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T00:33:07.550 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:33:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:33:07.264+0000 7f7ab1b08700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:33:07.555 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:07 smithi123 ceph-mon[31487]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:33:07.555 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:07 smithi123 ceph-mon[31487]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:33:07.555 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:07.555 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:07 smithi123 ceph-mon[31487]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:07.555 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:07 smithi123 ceph-mon[31487]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:07 smithi177 ceph-mon[36034]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:33:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:07 smithi177 ceph-mon[36034]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:33:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:07 smithi177 ceph-mon[36034]: from='osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:07.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:07 smithi177 ceph-mon[36034]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:33:08.340 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:08 smithi123 ceph-mon[31487]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:08.340 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:08 smithi123 ceph-mon[31487]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:33:08.340 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:08 smithi123 ceph-mon[31487]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:33:08.340 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:08 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:08 smithi204 ceph-mon[36345]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:08 smithi204 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:33:08.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:08 smithi204 ceph-mon[36345]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:33:08.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:08 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:08 smithi177 ceph-mon[36034]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:08 smithi177 ceph-mon[36034]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:33:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:08 smithi177 ceph-mon[36034]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:33:08.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:08 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0342652 s, 306 MB/s 2024-03-24T00:33:09.166 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] boot 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] boot 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:33:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:33:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: osd.8 [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] boot 2024-03-24T00:33:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:33:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:33:10.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_1 2024-03-24T00:33:10.542 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:33:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:33:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi204 2024-03-24T00:33:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi204 to 5693M 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi204 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi204 to 5693M 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:33:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi204 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi204 to 5693M 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:10.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:10.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:11.548 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:11 smithi123 ceph-mon[31487]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:33:11.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:11 smithi123 ceph-mon[31487]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:33:11.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:11 smithi123 ceph-mon[31487]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:11 smithi204 ceph-mon[36345]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:33:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:11 smithi204 ceph-mon[36345]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:33:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:11 smithi204 ceph-mon[36345]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:11 smithi177 ceph-mon[36034]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:33:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:11 smithi177 ceph-mon[36034]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:33:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:11 smithi177 ceph-mon[36034]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:12.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.768+0000 7f0bc5c00700 1 -- 172.21.15.123:0/319587215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105fa0 msgr2=0x7f0bc010c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.768+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/319587215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105fa0 0x7f0bc010c850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800b0d0 tx=0x7f0bb800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 -- 172.21.15.123:0/319587215 shutdown_connections 2024-03-24T00:33:12.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/319587215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105fa0 0x7f0bc010c850 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/319587215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc0104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/319587215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105600 0x7f0bc0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 -- 172.21.15.123:0/319587215 >> 172.21.15.123:0/319587215 conn(0x7f0bc00ff9a0 msgr2=0x7f0bc0101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 -- 172.21.15.123:0/319587215 shutdown_connections 2024-03-24T00:33:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.769+0000 7f0bc5c00700 1 -- 172.21.15.123:0/319587215 wait complete. 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 Processor -- start 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 -- start start 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105fa0 0x7f0bc01a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc010ffe0 con 0x7f0bc0105600 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc010fe60 con 0x7f0bc0105fa0 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.770+0000 7f0bc5c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0110160 con 0x7f0bc0104400 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41720/0 (socket says 172.21.15.123:41720) 2024-03-24T00:33:12.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 -- 172.21.15.123:0/2154860552 learned_addr learned my addr 172.21.15.123:0/2154860552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbea00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbf400700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105fa0 0x7f0bc01a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 -- 172.21.15.123:0/2154860552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 msgr2=0x7f0bc019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 -- 172.21.15.123:0/2154860552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105fa0 msgr2=0x7f0bc01a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105fa0 0x7f0bc01a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 -- 172.21.15.123:0/2154860552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc01a21a0 con 0x7f0bc0105600 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbea00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.771+0000 7f0bbe000700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9d0 tx=0x7f0bac00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.772+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00cbd0 con 0x7f0bc0105600 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.772+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc01a2490 con 0x7f0bc0105600 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.772+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bac00ee70 con 0x7f0bc0105600 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.772+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac018680 con 0x7f0bc0105600 2024-03-24T00:33:12.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.772+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc01a2a20 con 0x7f0bc0105600 2024-03-24T00:33:12.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.773+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0068e50 con 0x7f0bc0105600 2024-03-24T00:33:12.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.775+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0bac0188e0 con 0x7f0bc0105600 2024-03-24T00:33:12.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.776+0000 7f0bb7400700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 0x7f0ba40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.776+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f0bac0ce260 con 0x7f0bc0105600 2024-03-24T00:33:12.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.777+0000 7f0bbea00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 0x7f0ba40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.777+0000 7f0bbea00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 0x7f0ba40a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0105460 tx=0x7f0bb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:12.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:12.781+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0bac09bcb0 con 0x7f0bc0105600 2024-03-24T00:33:13.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:13.011+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bc00625d0 con 0x7f0ba40a2910 2024-03-24T00:33:13.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:12 smithi177 ceph-mon[36034]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:33:13.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:12 smithi204 ceph-mon[36345]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:33:13.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:12 smithi123 ceph-mon[31487]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:33:14.241 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:13 smithi123 ceph-mon[31487]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:14.242 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:14.242 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:13 smithi123 ceph-mon[31487]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:14.242 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:14.242 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:13 smithi204 ceph-mon[36345]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:13 smithi204 ceph-mon[36345]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:14.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:14.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:13 smithi177 ceph-mon[36034]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:14.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:13 smithi177 ceph-mon[36034]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:16 smithi123 ceph-mon[31487]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-24T00:33:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:16 smithi177 ceph-mon[36034]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-24T00:33:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:16 smithi204 ceph-mon[36345]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-24T00:33:17.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:17 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]: dispatch 2024-03-24T00:33:17.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:17 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]': finished 2024-03-24T00:33:17.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:17 smithi123 ceph-mon[31487]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:33:17.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:17.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:17 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]: dispatch 2024-03-24T00:33:17.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:17 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]': finished 2024-03-24T00:33:17.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:17 smithi177 ceph-mon[36034]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:33:17.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:17 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]: dispatch 2024-03-24T00:33:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:17 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1735134897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234e47dd-50a2-4522-8559-5cbf492d88d9"}]': finished 2024-03-24T00:33:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:17 smithi204 ceph-mon[36345]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:33:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:18.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:18 smithi177 ceph-mon[36034]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-24T00:33:18.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:18 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1819645201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:18 smithi204 ceph-mon[36345]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-24T00:33:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:18 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1819645201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:18 smithi123 ceph-mon[31487]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-24T00:33:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:18 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1819645201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:20.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:20 smithi177 ceph-mon[36034]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:33:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:20 smithi204 ceph-mon[36345]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:33:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:20 smithi123 ceph-mon[31487]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:33:22.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:22 smithi123 ceph-mon[31487]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-24T00:33:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:22 smithi177 ceph-mon[36034]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-24T00:33:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:22 smithi204 ceph-mon[36345]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-24T00:33:24.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:24 smithi177 ceph-mon[36034]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:33:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:24 smithi204 ceph-mon[36345]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:33:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:24 smithi123 ceph-mon[31487]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:33:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:26 smithi177 ceph-mon[36034]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:26 smithi204 ceph-mon[36345]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:26 smithi123 ceph-mon[31487]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:27.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:27 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:33:27.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:27 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:27.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:27 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:33:27.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:27 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:27.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:33:27.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:27 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:28.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:28 smithi123 ceph-mon[31487]: Deploying daemon osd.9 on smithi123 2024-03-24T00:33:28.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:28 smithi123 ceph-mon[31487]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:28 smithi177 ceph-mon[36034]: Deploying daemon osd.9 on smithi123 2024-03-24T00:33:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:28 smithi177 ceph-mon[36034]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:28 smithi204 ceph-mon[36345]: Deploying daemon osd.9 on smithi123 2024-03-24T00:33:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:28 smithi204 ceph-mon[36345]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:30 smithi177 ceph-mon[36034]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:30.532 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:30 smithi123 ceph-mon[31487]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:30 smithi204 ceph-mon[36345]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:32.313 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:32 smithi123 ceph-mon[31487]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:32.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:32 smithi177 ceph-mon[36034]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:32 smithi204 ceph-mon[36345]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:34 smithi177 ceph-mon[36034]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:34 smithi204 ceph-mon[36345]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:34.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:34 smithi123 ceph-mon[31487]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:36.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:36 smithi123 ceph-mon[31487]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:36.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:36 smithi177 ceph-mon[36034]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:36 smithi204 ceph-mon[36345]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:38.263 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:38 smithi123 ceph-mon[31487]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:38 smithi177 ceph-mon[36034]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:38 smithi204 ceph-mon[36345]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:39.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.856+0000 7f0bb7400700 1 -- 172.21.15.123:0/2154860552 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0bc00625d0 con 0x7f0ba40a2910 2024-03-24T00:33:39.858 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 9 on host 'smithi123' 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 msgr2=0x7f0ba40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 0x7f0ba40a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0105460 tx=0x7f0bb0009260 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 msgr2=0x7f0bc019aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9d0 tx=0x7f0bac00ece0 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 shutdown_connections 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0ba40a2910 0x7f0ba40a4dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc0105fa0 0x7f0bc01a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0bc0104400 0x7f0bc019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 --2- 172.21.15.123:0/2154860552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0105600 0x7f0bc019aa80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.859+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 >> 172.21.15.123:0/2154860552 conn(0x7f0bc00ff9a0 msgr2=0x7f0bc0108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.860+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 shutdown_connections 2024-03-24T00:33:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:33:39.860+0000 7f0bc5c00700 1 -- 172.21.15.123:0/2154860552 wait complete. 2024-03-24T00:33:40.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:33:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:39 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:40.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:33:40.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:39 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:40.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:33:40.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:39 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:40.612 DEBUG:teuthology.orchestra.run.smithi123:osd.9> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T00:33:40.619 INFO:tasks.cephadm:Deploying osd.10 on smithi177 with /dev/vg_nvme/lv_1... 2024-03-24T00:33:40.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:33:40.865 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:33:41.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:40 smithi177 ceph-mon[36034]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:33:41.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:40 smithi177 ceph-mon[36034]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:33:41.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:40 smithi177 ceph-mon[36034]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:33:41.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:40 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:41.219 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-mon[31487]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:33:41.219 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-mon[31487]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:33:41.219 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-mon[31487]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:33:41.219 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:41.220 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:33:40.933+0000 7f8ec6c00700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T00:33:41.220 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:33:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:33:40.955+0000 7f8ec1a00700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:33:41.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:40 smithi204 ceph-mon[36345]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:33:41.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:40 smithi204 ceph-mon[36345]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:33:41.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:40 smithi204 ceph-mon[36345]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-24T00:33:41.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:40 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:33:42.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:33:42.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:42.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.267 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:33:42.267 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:33:42.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:42.268 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: from='osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-24T00:33:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:33:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:33:42.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111609 s, 940 MB/s 2024-03-24T00:33:42.908 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-24T00:33:43.163 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:42 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:33:43.163 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:42 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:33:43.163 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:42 smithi123 ceph-mon[31487]: osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] boot 2024-03-24T00:33:43.163 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:42 smithi123 ceph-mon[31487]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:33:43.163 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:42 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:33:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:42 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:33:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:42 smithi177 ceph-mon[36034]: osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] boot 2024-03-24T00:33:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:42 smithi177 ceph-mon[36034]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:33:43.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:43.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:42 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:33:43.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:42 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:33:43.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:42 smithi204 ceph-mon[36345]: osd.9 [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] boot 2024-03-24T00:33:43.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:42 smithi204 ceph-mon[36345]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:33:43.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:33:43.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-03-24T00:33:43.766 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:33:44.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:43 smithi177 ceph-mon[36034]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:33:44.058 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:43 smithi177 ceph-mon[36034]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:44.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:43 smithi204 ceph-mon[36345]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:33:44.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:43 smithi204 ceph-mon[36345]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:44.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:43 smithi123 ceph-mon[31487]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:33:44.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:43 smithi123 ceph-mon[31487]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:45.117 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 -- 172.21.15.177:0/2093331843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c107f30 msgr2=0x7f3b1c10a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.117 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 --2- 172.21.15.177:0/2093331843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c107f30 0x7f3b1c10a320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400b0a0 tx=0x7f3b1400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 -- 172.21.15.177:0/2093331843 shutdown_connections 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 --2- 172.21.15.177:0/2093331843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c107f30 0x7f3b1c10a320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 --2- 172.21.15.177:0/2093331843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c1050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 --2- 172.21.15.177:0/2093331843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c105600 0x7f3b1c1079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 -- 172.21.15.177:0/2093331843 >> 172.21.15.177:0/2093331843 conn(0x7f3b1c0fc660 msgr2=0x7f3b1c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 -- 172.21.15.177:0/2093331843 shutdown_connections 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 -- 172.21.15.177:0/2093331843 wait complete. 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.115+0000 7f3b22626700 1 Processor -- start 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 -- start start 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c107f30 0x7f3b1c1011c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10de10 con 0x7f3b1c107f30 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10dc90 con 0x7f3b1c105600 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b22626700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10df90 con 0x7f3b1c102cd0 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:55344/0 (socket says 172.21.15.177:55344) 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 -- 172.21.15.177:0/911224013 learned_addr learned my addr 172.21.15.177:0/911224013 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1bfff700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 -- 172.21.15.177:0/911224013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 msgr2=0x7f3b1c102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 -- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c107f30 msgr2=0x7f3b1c1011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c107f30 0x7f3b1c1011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 -- 172.21.15.177:0/911224013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c101a70 con 0x7f3b1c105600 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1bfff700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.116+0000 7f3b1b7fe700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c970 tx=0x7f3b1000cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.117+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10007aa0 con 0x7f3b1c105600 2024-03-24T00:33:45.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.117+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c1a25f0 con 0x7f3b1c105600 2024-03-24T00:33:45.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.117+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b10004d10 con 0x7f3b1c105600 2024-03-24T00:33:45.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.117+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10005690 con 0x7f3b1c105600 2024-03-24T00:33:45.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.117+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c1a2b00 con 0x7f3b1c105600 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.118+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c068e50 con 0x7f3b1c105600 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.119+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3b10004750 con 0x7f3b1c105600 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.120+0000 7f3b197fa700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 0x7f3b040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.120+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f3b100ce3a0 con 0x7f3b1c105600 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.120+0000 7f3b1bfff700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 0x7f3b040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.121+0000 7f3b1bfff700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 0x7f3b040a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c007900 tx=0x7f3b0c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.124+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b1009bd60 con 0x7f3b1c105600 2024-03-24T00:33:45.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:33:45.329+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b1c0625d0 con 0x7f3b040a2890 2024-03-24T00:33:46.305 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:46.305 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:46.305 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:46.306 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.306 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.306 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.306 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.306 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.307 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.308 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:46.308 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.308 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.308 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:46 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:46.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:46 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:46.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:33:46.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:33:46.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:46.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:46 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:33:47.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:47 smithi177 ceph-mon[36034]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:47.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:47 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi123 2024-03-24T00:33:47.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:47 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi123 to 4272M 2024-03-24T00:33:47.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:47 smithi123 ceph-mon[31487]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:47.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:47 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi123 2024-03-24T00:33:47.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:47 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi123 to 4272M 2024-03-24T00:33:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:47 smithi204 ceph-mon[36345]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:33:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:47 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi123 2024-03-24T00:33:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:47 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi123 to 4272M 2024-03-24T00:33:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:48.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/3159131362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]': finished 2024-03-24T00:33:48.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:33:48.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:48 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:33:48.411 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:48.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/3159131362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]': finished 2024-03-24T00:33:48.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:33:48.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:48 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:33:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/3159131362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]: dispatch 2024-03-24T00:33:48.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52997bce-ef96-4fbd-803e-2c505beda096"}]': finished 2024-03-24T00:33:48.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:33:48.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:48 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:33:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:49 smithi177 ceph-mon[36034]: from='client.? 172.21.15.177:0/3693148839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:49 smithi204 ceph-mon[36345]: from='client.? 172.21.15.177:0/3693148839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:49 smithi123 ceph-mon[31487]: from='client.? 172.21.15.177:0/3693148839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:33:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:50 smithi177 ceph-mon[36034]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:50 smithi204 ceph-mon[36345]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:50 smithi123 ceph-mon[31487]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:52.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:52 smithi177 ceph-mon[36034]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:52 smithi204 ceph-mon[36345]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:52 smithi123 ceph-mon[31487]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:54.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:54 smithi123 ceph-mon[31487]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:54.388 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:54 smithi204 ceph-mon[36345]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:54 smithi177 ceph-mon[36034]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:56.451 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:56 smithi177 ceph-mon[36034]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:56.451 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:33:56.451 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:56 smithi204 ceph-mon[36345]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:33:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:56 smithi123 ceph-mon[31487]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:33:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:57.321 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:57 smithi177 ceph-mon[36034]: Deploying daemon osd.10 on smithi177 2024-03-24T00:33:57.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:57 smithi204 ceph-mon[36345]: Deploying daemon osd.10 on smithi177 2024-03-24T00:33:57.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:57 smithi123 ceph-mon[31487]: Deploying daemon osd.10 on smithi177 2024-03-24T00:33:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:33:58 smithi177 ceph-mon[36034]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:58.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:33:58 smithi204 ceph-mon[36345]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:33:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:33:58 smithi123 ceph-mon[31487]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:00.404 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:00 smithi177 ceph-mon[36034]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:00 smithi204 ceph-mon[36345]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:00 smithi123 ceph-mon[31487]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:02 smithi204 ceph-mon[36345]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:02 smithi123 ceph-mon[31487]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:02 smithi177 ceph-mon[36034]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:04.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 10 on host 'smithi177' 2024-03-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.061+0000 7f3b197fa700 1 -- 172.21.15.177:0/911224013 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3b1c0625d0 con 0x7f3b040a2890 2024-03-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 msgr2=0x7f3b040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 0x7f3b040a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c007900 tx=0x7f3b0c008040 comp rx=0 tx=0).stop 2024-03-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 msgr2=0x7f3b1c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c970 tx=0x7f3b1000cd30 comp rx=0 tx=0).stop 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 shutdown_connections 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3b040a2890 0x7f3b040a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c107f30 0x7f3b1c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b1c102cd0 0x7f3b1c102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 --2- 172.21.15.177:0/911224013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b1c105600 0x7f3b1c100c80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 >> 172.21.15.177:0/911224013 conn(0x7f3b1c0fc660 msgr2=0x7f3b1c103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 shutdown_connections 2024-03-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-24T00:34:05.064+0000 7f3b22626700 1 -- 172.21.15.177:0/911224013 wait complete. 2024-03-24T00:34:05.692 DEBUG:teuthology.orchestra.run.smithi177:osd.10> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T00:34:05.694 INFO:tasks.cephadm:Deploying osd.11 on smithi204 with /dev/vg_nvme/lv_1... 2024-03-24T00:34:05.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:34:05.936 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:34:06.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:06.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:06.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:06.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:06 smithi177 ceph-mon[36034]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:06.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:06 smithi204 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:06.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:06.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:06.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:06 smithi123 ceph-mon[31487]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:34:07.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:07 smithi204 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:34:07.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:07 smithi204 ceph-mon[36345]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:07 smithi204 ceph-mon[36345]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:34:07.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:07.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:07 smithi204 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:07 smithi123 ceph-mon[31487]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:34:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:07 smithi123 ceph-mon[31487]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:07 smithi123 ceph-mon[31487]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:34:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:07.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:07 smithi123 ceph-mon[31487]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-mon[36034]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:34:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-mon[36034]: from='osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-mon[36034]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:34:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-mon[36034]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112701 s, 930 MB/s 2024-03-24T00:34:07.917 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-24T00:34:08.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-mon[36034]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:34:08.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-mon[36034]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:08.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-mon[36034]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:34:08.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:08.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-mon[36034]: from='osd.10 ' entity='osd.10' 2024-03-24T00:34:08.213 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:34:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:34:07.943+0000 7fd3f2122700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T00:34:08.213 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:34:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:34:08.054+0000 7fd3e970c700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:34:08.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:08 smithi204 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:34:08.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:08 smithi204 ceph-mon[36345]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:08.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:08 smithi204 ceph-mon[36345]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:34:08.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:08 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:08.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:08 smithi204 ceph-mon[36345]: from='osd.10 ' entity='osd.10' 2024-03-24T00:34:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:08 smithi123 ceph-mon[31487]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-24T00:34:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:08 smithi123 ceph-mon[31487]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:08 smithi123 ceph-mon[31487]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:34:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:08 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:08.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:08 smithi123 ceph-mon[31487]: from='osd.10 ' entity='osd.10' 2024-03-24T00:34:08.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-03-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:34:09.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:34:09.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: osdmap e73: 11 total, 10 up, 11 in 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:09.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] boot 2024-03-24T00:34:09.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:34:09.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: osdmap e73: 11 total, 10 up, 11 in 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] boot 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:34:09.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:09.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: osdmap e73: 11 total, 10 up, 11 in 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:09.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:09.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:09.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: osd.10 [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] boot 2024-03-24T00:34:09.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:34:09.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:34:10.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:10 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi177 2024-03-24T00:34:10.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:10 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-24T00:34:10.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:10 smithi204 ceph-mon[36345]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:10.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:10 smithi204 ceph-mon[36345]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:34:10.348 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 -- 172.21.15.204:0/3730092359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b4102c60 msgr2=0x7fe3b4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.348 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/3730092359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b4102c60 0x7fe3b4109510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00b0a0 tx=0x7fe3ac00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.348 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 -- 172.21.15.204:0/3730092359 shutdown_connections 2024-03-24T00:34:10.348 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/3730092359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b4102c60 0x7fe3b4109510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.348 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/3730092359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41010c0 0x7fe3b41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/3730092359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41022c0 0x7fe3b4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.347+0000 7fe3ba40f700 1 -- 172.21.15.204:0/3730092359 >> 172.21.15.204:0/3730092359 conn(0x7fe3b40fc680 msgr2=0x7fe3b40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 -- 172.21.15.204:0/3730092359 shutdown_connections 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 -- 172.21.15.204:0/3730092359 wait complete. 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 Processor -- start 2024-03-24T00:34:10.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 -- start start 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 0x7fe3b41977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.348+0000 7fe3ba40f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102c60 0x7fe3b419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3ba40f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410cd00 con 0x7fe3b4102c60 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3ba40f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410cb80 con 0x7fe3b41022c0 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3ba40f700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410ce80 con 0x7fe3b41010c0 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:39810/0 (socket says 172.21.15.204:39810) 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 -- 172.21.15.204:0/1443458370 learned_addr learned my addr 172.21.15.204:0/1443458370 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:34:10.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b37fe700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 0x7fe3b41977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 -- 172.21.15.204:0/1443458370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 msgr2=0x7fe3b41977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 0x7fe3b41977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 -- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102c60 msgr2=0x7fe3b419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102c60 0x7fe3b419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 -- 172.21.15.204:0/1443458370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b419eef0 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b37fe700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 0x7fe3b41977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b3fff700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4007b80 tx=0x7fe3a4007f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4019020 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a4004500 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b406a930 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a400e600 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.349+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b406aef0 con 0x7fe3b41010c0 2024-03-24T00:34:10.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.350+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4068e50 con 0x7fe3b41010c0 2024-03-24T00:34:10.353 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.351+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe3a4004000 con 0x7fe3b41010c0 2024-03-24T00:34:10.353 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.352+0000 7fe3b17fa700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 0x7fe39c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.353 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.352+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fe3a40cf900 con 0x7fe3b41010c0 2024-03-24T00:34:10.353 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.352+0000 7fe3b37fe700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 0x7fe39c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.353 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.353+0000 7fe3b37fe700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 0x7fe39c0a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8007900 tx=0x7fe3a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.356+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe3a4023020 con 0x7fe3b41010c0 2024-03-24T00:34:10.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:10 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi177 2024-03-24T00:34:10.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:10 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-24T00:34:10.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:10 smithi123 ceph-mon[31487]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:10.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:10 smithi123 ceph-mon[31487]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:34:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:10 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi177 2024-03-24T00:34:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:10 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-24T00:34:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:10 smithi177 ceph-mon[36034]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:34:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:10 smithi177 ceph-mon[36034]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:34:10.564 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:10.564+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3b406b150 con 0x7fe39c0a2890 2024-03-24T00:34:11.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:34:11.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:34:11.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:11.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:11 smithi204 ceph-mon[36345]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:34:11.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:34:11.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:34:11.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:11.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:11 smithi123 ceph-mon[31487]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:34:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:34:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:34:11.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:11.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:11 smithi177 ceph-mon[36034]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:34:12.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:12 smithi204 ceph-mon[36345]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:12.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:12 smithi204 ceph-mon[36345]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:12 smithi123 ceph-mon[31487]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:12.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:12 smithi123 ceph-mon[31487]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:12 smithi177 ceph-mon[36034]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:12 smithi177 ceph-mon[36034]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:13.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:13 smithi123 ceph-mon[31487]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:34:13.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:13 smithi177 ceph-mon[36034]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:34:13.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:13 smithi204 ceph-mon[36345]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:34:14.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: osdmap e78: 11 total, 11 up, 11 in 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/2620177582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]': finished 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:14.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:14 smithi177 ceph-mon[36034]: from='client.? 172.21.15.204:0/1654808494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:34:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: osdmap e78: 11 total, 11 up, 11 in 2024-03-24T00:34:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/2620177582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]': finished 2024-03-24T00:34:14.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:34:14.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:14.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:14 smithi204 ceph-mon[36345]: from='client.? 172.21.15.204:0/1654808494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:34:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: osdmap e78: 11 total, 11 up, 11 in 2024-03-24T00:34:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/2620177582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]: dispatch 2024-03-24T00:34:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233d66e6-e30c-427c-b37e-dda971c1b299"}]': finished 2024-03-24T00:34:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:34:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:14 smithi123 ceph-mon[31487]: from='client.? 172.21.15.204:0/1654808494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:34:16.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:16 smithi177 ceph-mon[36034]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:34:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:16 smithi204 ceph-mon[36345]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:34:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:16 smithi123 ceph-mon[31487]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:34:18.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:18 smithi204 ceph-mon[36345]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:34:18.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:18 smithi177 ceph-mon[36034]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:34:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:18 smithi123 ceph-mon[31487]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:34:20.417 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:20 smithi204 ceph-mon[36345]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-24T00:34:20.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:20 smithi177 ceph-mon[36034]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-24T00:34:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:20 smithi123 ceph-mon[31487]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-24T00:34:21.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:34:21.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:21.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:34:21.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:21.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:34:21.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:22.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:22 smithi204 ceph-mon[36345]: Deploying daemon osd.11 on smithi204 2024-03-24T00:34:22.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:22 smithi204 ceph-mon[36345]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:34:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:22 smithi177 ceph-mon[36034]: Deploying daemon osd.11 on smithi204 2024-03-24T00:34:22.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:22 smithi177 ceph-mon[36034]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:34:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:22 smithi123 ceph-mon[31487]: Deploying daemon osd.11 on smithi204 2024-03-24T00:34:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:22 smithi123 ceph-mon[31487]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:34:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:24 smithi204 ceph-mon[36345]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:34:24.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:24 smithi177 ceph-mon[36034]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:34:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:24 smithi123 ceph-mon[31487]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:34:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:26 smithi177 ceph-mon[36034]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-24T00:34:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:26.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:26.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:26 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:26 smithi204 ceph-mon[36345]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-24T00:34:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:26 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:26 smithi123 ceph-mon[31487]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-24T00:34:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:26.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:26.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:26 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:28.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:28 smithi204 ceph-mon[36345]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:28 smithi177 ceph-mon[36034]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:28 smithi123 ceph-mon[31487]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:29.700 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 11 on host 'smithi204' 2024-03-24T00:34:29.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.699+0000 7fe3b17fa700 1 -- 172.21.15.204:0/1443458370 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe3b406b150 con 0x7fe39c0a2890 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 msgr2=0x7fe39c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 0x7fe39c0a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8007900 tx=0x7fe3a8008040 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 msgr2=0x7fe3b4197290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4007b80 tx=0x7fe3a4007f40 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 shutdown_connections 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe39c0a2890 0x7fe39c0a4d50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102c60 0x7fe3b419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3b41010c0 0x7fe3b4197290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 --2- 172.21.15.204:0/1443458370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3b41022c0 0x7fe3b41977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 >> 172.21.15.204:0/1443458370 conn(0x7fe3b40fc680 msgr2=0x7fe3b4105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 shutdown_connections 2024-03-24T00:34:29.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:29.701+0000 7fe3ba40f700 1 -- 172.21.15.204:0/1443458370 wait complete. 2024-03-24T00:34:30.001 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.001 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.001 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.002 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:29 smithi204 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:29 smithi123 ceph-mon[31487]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:30.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:29 smithi177 ceph-mon[36034]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:34:30.327 DEBUG:teuthology.orchestra.run.smithi204:osd.11> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T00:34:30.330 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T00:34:30.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T00:34:30.780 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:31.962 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:31 smithi123 ceph-mon[31487]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:31.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:31 smithi177 ceph-mon[36034]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:32.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:34:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: from='osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:34:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-24T00:34:32.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:31 smithi204 ceph-mon[36345]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:34:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-mon[36345]: osdmap e81: 12 total, 11 up, 12 in 2024-03-24T00:34:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:32.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-mon[36345]: from='osd.11 ' entity='osd.11' 2024-03-24T00:34:32.822 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:34:32.571+0000 7fd6a0355700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T00:34:32.822 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:34:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:34:32.704+0000 7fd698140700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:34:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:32 smithi123 ceph-mon[31487]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:34:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:32 smithi123 ceph-mon[31487]: osdmap e81: 12 total, 11 up, 12 in 2024-03-24T00:34:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:32 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:32 smithi123 ceph-mon[31487]: from='osd.11 ' entity='osd.11' 2024-03-24T00:34:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:32 smithi177 ceph-mon[36034]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-24T00:34:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:32 smithi177 ceph-mon[36034]: osdmap e81: 12 total, 11 up, 12 in 2024-03-24T00:34:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:32 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:32 smithi177 ceph-mon[36034]: from='osd.11 ' entity='osd.11' 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 -- 172.21.15.123:0/2042143207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c107ea0 msgr2=0x7f5f2c10a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 --2- 172.21.15.123:0/2042143207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c10a310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400b0a0 tx=0x7f5f2400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 -- 172.21.15.123:0/2042143207 shutdown_connections 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 --2- 172.21.15.123:0/2042143207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c10a310 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 --2- 172.21.15.123:0/2042143207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c102c40 0x7f5f2c105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 --2- 172.21.15.123:0/2042143207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 -- 172.21.15.123:0/2042143207 >> 172.21.15.123:0/2042143207 conn(0x7f5f2c0fc650 msgr2=0x7f5f2c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 -- 172.21.15.123:0/2042143207 shutdown_connections 2024-03-24T00:34:33.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.105+0000 7f5f32400700 1 -- 172.21.15.123:0/2042143207 wait complete. 2024-03-24T00:34:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.106+0000 7f5f32400700 1 Processor -- start 2024-03-24T00:34:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 -- start start 2024-03-24T00:34:33.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c1a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c10dde0 con 0x7f5f2c102c40 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c10dc60 con 0x7f5f2c107ea0 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f32400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c10df60 con 0x7f5f2c105570 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f2b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f2b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60474/0 (socket says 172.21.15.123:60474) 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f2b400700 1 -- 172.21.15.123:0/137539832 learned_addr learned my addr 172.21.15.123:0/137539832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f2aa00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.107+0000 7f5f2be00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c1a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 -- 172.21.15.123:0/137539832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 msgr2=0x7f5f2c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 -- 172.21.15.123:0/137539832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c107ea0 msgr2=0x7f5f2c1a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c1a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 -- 172.21.15.123:0/137539832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c1a2220 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2aa00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f2b400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00d980 tx=0x7f5f1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c016070 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c1a2510 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f1c009db0 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.108+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00f3c0 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.109+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c1a2aa0 con 0x7f5f2c102c40 2024-03-24T00:34:33.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.110+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c068e50 con 0x7f5f2c102c40 2024-03-24T00:34:33.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.111+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5f1c019070 con 0x7f5f2c102c40 2024-03-24T00:34:33.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.113+0000 7f5f23e00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 0x7f5f100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.113+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f5f1c0cdff0 con 0x7f5f2c102c40 2024-03-24T00:34:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.113+0000 7f5f2aa00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 0x7f5f100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.114+0000 7f5f2aa00700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 0x7f5f100a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f18011490 tx=0x7f5f18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.118+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f1c09b5d0 con 0x7f5f2c102c40 2024-03-24T00:34:33.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.351+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f5f2c062da0 con 0x7f5f2c102c40 2024-03-24T00:34:33.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.352+0000 7f5f23e00700 1 -- 172.21.15.123:0/137539832 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f5f1c09aed0 con 0x7f5f2c102c40 2024-03-24T00:34:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 msgr2=0x7f5f100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 0x7f5f100a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f18011490 tx=0x7f5f18009260 comp rx=0 tx=0).stop 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 msgr2=0x7f5f2c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00d980 tx=0x7f5f1c00dc90 comp rx=0 tx=0).stop 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 shutdown_connections 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5f100a2890 0x7f5f100a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f2c107ea0 0x7f5f2c1a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f2c102c40 0x7f5f2c19a5c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 --2- 172.21.15.123:0/137539832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f2c105570 0x7f5f2c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:33.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 >> 172.21.15.123:0/137539832 conn(0x7f5f2c0fc650 msgr2=0x7f5f2c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:33.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 shutdown_connections 2024-03-24T00:34:33.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:33.355+0000 7f5f32400700 1 -- 172.21.15.123:0/137539832 wait complete. 2024-03-24T00:34:33.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: purged_snaps scrub starts 2024-03-24T00:34:33.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: purged_snaps scrub ok 2024-03-24T00:34:33.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: Detected new or changed devices on smithi204 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi204 to 4270M 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:33 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/137539832' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:33.977 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: purged_snaps scrub starts 2024-03-24T00:34:33.977 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: purged_snaps scrub ok 2024-03-24T00:34:33.977 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:33.977 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: Detected new or changed devices on smithi204 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi204 to 4270M 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:33.978 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:33 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/137539832' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:34.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: purged_snaps scrub starts 2024-03-24T00:34:34.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: purged_snaps scrub ok 2024-03-24T00:34:34.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: Detected new or changed devices on smithi204 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi204 to 4270M 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:34.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:33 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/137539832' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:34.286 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1711240449,"num_in_osds":12,"osd_in_since":1711240453,"num_remapped_pgs":0} 2024-03-24T00:34:34.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:34.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:34 smithi177 ceph-mon[36034]: osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] boot 2024-03-24T00:34:34.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:34 smithi177 ceph-mon[36034]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:34:34.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:35.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:35.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:34 smithi204 ceph-mon[36345]: osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] boot 2024-03-24T00:34:35.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:34 smithi204 ceph-mon[36345]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:34:35.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:35.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:35.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:34 smithi123 ceph-mon[31487]: osd.11 [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] boot 2024-03-24T00:34:35.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:34 smithi123 ceph-mon[31487]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:34:35.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:34:35.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T00:34:35.728 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:34:35.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:35 smithi177 ceph-mon[36034]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:34:35.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:35 smithi177 ceph-mon[36034]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:36.044 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:35 smithi123 ceph-mon[31487]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:34:36.045 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:35 smithi123 ceph-mon[31487]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:36.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:35 smithi204 ceph-mon[36345]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:34:36.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:35 smithi204 ceph-mon[36345]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.953+0000 7f19a8000700 1 -- 172.21.15.123:0/2928766804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0102c50 msgr2=0x7f19a0105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.953+0000 7f19a8000700 1 --2- 172.21.15.123:0/2928766804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0102c50 0x7f19a0105040 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1994009990 tx=0x7f1994009ca0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 -- 172.21.15.123:0/2928766804 shutdown_connections 2024-03-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 --2- 172.21.15.123:0/2928766804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0107eb0 0x7f19a010a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 --2- 172.21.15.123:0/2928766804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0102c50 0x7f19a0105040 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 --2- 172.21.15.123:0/2928766804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0105580 0x7f19a0107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 -- 172.21.15.123:0/2928766804 >> 172.21.15.123:0/2928766804 conn(0x7f19a00fc660 msgr2=0x7f19a00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 -- 172.21.15.123:0/2928766804 shutdown_connections 2024-03-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.954+0000 7f19a8000700 1 -- 172.21.15.123:0/2928766804 wait complete. 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 Processor -- start 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 -- start start 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0102c50 0x7f19a019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 0x7f19a019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010de20 con 0x7f19a0107eb0 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010dca0 con 0x7f19a0105580 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.955+0000 7f19a8000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010dfa0 con 0x7f19a0102c50 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a4c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 0x7f19a019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52676/0 (socket says 172.21.15.123:52676) 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a5600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0102c50 0x7f19a019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a4c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 0x7f19a019ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51752/0 (socket says 172.21.15.123:51752) 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 -- 172.21.15.123:0/555938828 learned_addr learned my addr 172.21.15.123:0/555938828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 -- 172.21.15.123:0/555938828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0102c50 msgr2=0x7f19a019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0102c50 0x7f19a019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 -- 172.21.15.123:0/555938828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 msgr2=0x7f19a019ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 0x7f19a019ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.956+0000 7f19a6000700 1 -- 172.21.15.123:0/555938828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a01a2330 con 0x7f19a0107eb0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f19a6000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f199800efc0 tx=0x7f199800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19980092e0 con 0x7f19a0107eb0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1998004bb0 con 0x7f19a0107eb0 2024-03-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a01a25c0 con 0x7f19a0107eb0 2024-03-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998004280 con 0x7f19a0107eb0 2024-03-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.957+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a01a2b50 con 0x7f19a0107eb0 2024-03-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.958+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0068e50 con 0x7f19a0107eb0 2024-03-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.960+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f199801b030 con 0x7f19a0107eb0 2024-03-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.961+0000 7f199e000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 0x7f19880a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.962+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f19980ce0b0 con 0x7f19a0107eb0 2024-03-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.962+0000 7f19a5600700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 0x7f19880a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.962+0000 7f19a5600700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 0x7f19880a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f199400b7b0 tx=0x7f199400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:37.966+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1998022020 con 0x7f19a0107eb0 2024-03-24T00:34:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.187+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f19a0062da0 con 0x7f19a0107eb0 2024-03-24T00:34:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.188+0000 7f199e000700 1 -- 172.21.15.123:0/555938828 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) v1 ==== 74+0+133 (secure 0 0 0) 0x7f1998010040 con 0x7f19a0107eb0 2024-03-24T00:34:38.188 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 msgr2=0x7f19880a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 0x7f19880a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f199400b7b0 tx=0x7f199400b880 comp rx=0 tx=0).stop 2024-03-24T00:34:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 msgr2=0x7f19a01a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f199800efc0 tx=0x7f199800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:34:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 shutdown_connections 2024-03-24T00:34:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f19880a2910 0x7f19880a4dd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19a0107eb0 0x7f19a01a1c90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a0102c50 0x7f19a019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 --2- 172.21.15.123:0/555938828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a0105580 0x7f19a019ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 >> 172.21.15.123:0/555938828 conn(0x7f19a00fc660 msgr2=0x7f19a01024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 shutdown_connections 2024-03-24T00:34:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:38.191+0000 7f19a8000700 1 -- 172.21.15.123:0/555938828 wait complete. 2024-03-24T00:34:38.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:38 smithi123 ceph-mon[31487]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:38 smithi177 ceph-mon[36034]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:38 smithi204 ceph-mon[36345]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:39.079 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1711240473,"num_in_osds":12,"osd_in_since":1711240453,"num_remapped_pgs":0} 2024-03-24T00:34:39.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:34:39.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:39 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/555938828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:39.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:39 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/555938828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:39.525 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:34:39.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:39 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/555938828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:34:40.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:40 smithi123 ceph-mon[31487]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:40 smithi177 ceph-mon[36034]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:40 smithi204 ceph-mon[36345]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 -- 172.21.15.123:0/2453147928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c1045e0 msgr2=0x7f0b1c1049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 --2- 172.21.15.123:0/2453147928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1049c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009920 tx=0x7f0b0c009c30 comp rx=0 tx=0).stop 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 -- 172.21.15.123:0/2453147928 shutdown_connections 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 --2- 172.21.15.123:0/2453147928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 --2- 172.21.15.123:0/2453147928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1049c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 --2- 172.21.15.123:0/2453147928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c104f90 0x7f0b1c108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 -- 172.21.15.123:0/2453147928 >> 172.21.15.123:0/2453147928 conn(0x7f0b1c0ffe70 msgr2=0x7f0b1c102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 -- 172.21.15.123:0/2453147928 shutdown_connections 2024-03-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.541+0000 7f0b22800700 1 -- 172.21.15.123:0/2453147928 wait complete. 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.542+0000 7f0b22800700 1 Processor -- start 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 -- start start 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 0x7f0b1c1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:41.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c1146a0 con 0x7f0b1c104f90 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c114520 con 0x7f0b1c1045e0 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b22800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c114820 con 0x7f0b1c1096c0 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1b400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 0x7f0b1c1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1b400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51778/0 (socket says 172.21.15.123:51778) 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1b400700 1 -- 172.21.15.123:0/37167644 learned_addr learned my addr 172.21.15.123:0/37167644 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 0x7f0b1c1a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52686/0 (socket says 172.21.15.123:52686) 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.543+0000 7f0b1be00700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 -- 172.21.15.123:0/37167644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 msgr2=0x7f0b1c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 -- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 msgr2=0x7f0b1c1a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 0x7f0b1c1a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 -- 172.21.15.123:0/37167644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c19b120 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1be00700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b1b400700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005e90 tx=0x7f0b0c00fa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0051c0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b0c00b560 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c19b3b0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.544+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c003bf0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.545+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c1a76d0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.546+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c105ac0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.547+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0b0c00baa0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.549+0000 7f0b13e00700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 0x7f0b000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:41.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.549+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0b0c0cf780 con 0x7f0b1c1045e0 2024-03-24T00:34:41.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.549+0000 7f0b1aa00700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 0x7f0b000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:41.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.550+0000 7f0b1aa00700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 0x7f0b000a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c10ac70 tx=0x7f0b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:41.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.551+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0b0c013030 con 0x7f0b1c1045e0 2024-03-24T00:34:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.771+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b1c02da30 con 0x7f0b1c1045e0 2024-03-24T00:34:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.772+0000 7f0b13e00700 1 -- 172.21.15.123:0/37167644 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18515 (secure 0 0 0) 0x7f0b0c09ccb0 con 0x7f0b1c1045e0 2024-03-24T00:34:41.774 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:41.774 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:34:34.673592+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.775+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 msgr2=0x7f0b000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.775+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 0x7f0b000a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c10ac70 tx=0x7f0b08009260 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 msgr2=0x7f0b1c1a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005e90 tx=0x7f0b0c00fa90 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 shutdown_connections 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b000a2890 0x7f0b000a4d50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b1c1096c0 0x7f0b1c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b1c1045e0 0x7f0b1c1a0800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 --2- 172.21.15.123:0/37167644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b1c104f90 0x7f0b1c1a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 >> 172.21.15.123:0/37167644 conn(0x7f0b1c0ffe70 msgr2=0x7f0b1c1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:41.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 shutdown_connections 2024-03-24T00:34:41.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:41.776+0000 7f0b22800700 1 -- 172.21.15.123:0/37167644 wait complete. 2024-03-24T00:34:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:42 smithi177 ceph-mon[36034]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/37167644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:42 smithi204 ceph-mon[36345]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/37167644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:42.573 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:42 smithi123 ceph-mon[31487]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:42.573 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/37167644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:42.632 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:34:42.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:34:44.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:44 smithi123 ceph-mon[31487]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:44.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:44 smithi177 ceph-mon[36034]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:44 smithi204 ceph-mon[36345]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 -- 172.21.15.123:0/86574558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b01096c0 msgr2=0x7f12b0111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 --2- 172.21.15.123:0/86574558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b01096c0 0x7f12b0111290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12a800b0d0 tx=0x7f12a800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 -- 172.21.15.123:0/86574558 shutdown_connections 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 --2- 172.21.15.123:0/86574558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b01096c0 0x7f12b0111290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 --2- 172.21.15.123:0/86574558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01045e0 0x7f12b01049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 --2- 172.21.15.123:0/86574558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b0104f90 0x7f12b0108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 -- 172.21.15.123:0/86574558 >> 172.21.15.123:0/86574558 conn(0x7f12b00ffe70 msgr2=0x7f12b0102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 -- 172.21.15.123:0/86574558 shutdown_connections 2024-03-24T00:34:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.184+0000 7f12b6c00700 1 -- 172.21.15.123:0/86574558 wait complete. 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.185+0000 7f12b6c00700 1 Processor -- start 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 -- start start 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 0x7f12b01a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01096c0 0x7f12b019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b01146d0 con 0x7f12b01096c0 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0114550 con 0x7f12b01045e0 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12b6c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0114850 con 0x7f12b0104f90 2024-03-24T00:34:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.186+0000 7f12afe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12b4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01096c0 0x7f12b019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51796/0 (socket says 172.21.15.123:51796) 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 -- 172.21.15.123:0/2653064376 learned_addr learned my addr 172.21.15.123:0/2653064376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12af400700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 0x7f12b01a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 -- 172.21.15.123:0/2653064376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 msgr2=0x7f12b01a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 0x7f12b01a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 -- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01096c0 msgr2=0x7f12b019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01096c0 0x7f12b019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 -- 172.21.15.123:0/2653064376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b019b010 con 0x7f12b01045e0 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12af400700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 0x7f12b01a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.187+0000 7f12afe00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12a000e9a0 tx=0x7f12a000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.188+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000ca10 con 0x7f12b01045e0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.188+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12a000cbb0 con 0x7f12b01045e0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.188+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0018680 con 0x7f12b01045e0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.188+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b019b300 con 0x7f12b01045e0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.188+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b01a75c0 con 0x7f12b01045e0 2024-03-24T00:34:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.189+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b0068e50 con 0x7f12b01045e0 2024-03-24T00:34:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.191+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f12a0010430 con 0x7f12b01045e0 2024-03-24T00:34:45.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.192+0000 7f12acc00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 0x7f12940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:45.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.192+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12a00ce810 con 0x7f12b01045e0 2024-03-24T00:34:45.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.192+0000 7f12af400700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 0x7f12940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:45.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.193+0000 7f12af400700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 0x7f12940a4dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12b010ac20 tx=0x7f129c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:45.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.197+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12a009bcb0 con 0x7f12b01045e0 2024-03-24T00:34:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.421+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12b002da30 con 0x7f12b01045e0 2024-03-24T00:34:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.422+0000 7f12acc00700 1 -- 172.21.15.123:0/2653064376 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12a009b5b0 con 0x7f12b01045e0 2024-03-24T00:34:45.422 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:34:45.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 msgr2=0x7f12940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:45.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 0x7f12940a4dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12b010ac20 tx=0x7f129c009260 comp rx=0 tx=0).stop 2024-03-24T00:34:45.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 msgr2=0x7f12b01a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:45.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12a000e9a0 tx=0x7f12a000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 shutdown_connections 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12940a2910 0x7f12940a4dd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b01096c0 0x7f12b019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12b01045e0 0x7f12b01a06f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 --2- 172.21.15.123:0/2653064376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12b0104f90 0x7f12b01a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.425+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 >> 172.21.15.123:0/2653064376 conn(0x7f12b00ffe70 msgr2=0x7f12b01009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.426+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 shutdown_connections 2024-03-24T00:34:45.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:45.426+0000 7f12b6c00700 1 -- 172.21.15.123:0/2653064376 wait complete. 2024-03-24T00:34:46.240 INFO:tasks.cephadm:Adding mds.a on smithi123 2024-03-24T00:34:46.240 INFO:tasks.cephadm:Adding mds.d on smithi123 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.g on smithi123 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.j on smithi123 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.b on smithi177 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.e on smithi177 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.h on smithi177 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.k on smithi177 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.c on smithi204 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.f on smithi204 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.i on smithi204 2024-03-24T00:34:46.241 INFO:tasks.cephadm:Adding mds.l on smithi204 2024-03-24T00:34:46.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l' 2024-03-24T00:34:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:46 smithi177 ceph-mon[36034]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:46 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2653064376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:46.493 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:34:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:46 smithi204 ceph-mon[36345]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:46 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2653064376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:46 smithi123 ceph-mon[31487]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:46 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2653064376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 -- 172.21.15.204:0/1883633948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c102c90 msgr2=0x7fa81c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 --2- 172.21.15.204:0/1883633948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c102c90 0x7fa81c109540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b0a0 tx=0x7fa81400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 -- 172.21.15.204:0/1883633948 shutdown_connections 2024-03-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 --2- 172.21.15.204:0/1883633948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c102c90 0x7fa81c109540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 --2- 172.21.15.204:0/1883633948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1010f0 0x7fa81c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 --2- 172.21.15.204:0/1883633948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa81c1022f0 0x7fa81c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 -- 172.21.15.204:0/1883633948 >> 172.21.15.204:0/1883633948 conn(0x7fa81c0fc690 msgr2=0x7fa81c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 -- 172.21.15.204:0/1883633948 shutdown_connections 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.044+0000 7fa8233ec700 1 -- 172.21.15.204:0/1883633948 wait complete. 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 Processor -- start 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 -- start start 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 0x7fa81c19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa81c102c90 0x7fa81c1a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10cdd0 con 0x7fa81c102c90 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10cc50 con 0x7fa81c1022f0 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa8233ec700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10cf50 con 0x7fa81c1010f0 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:50560/0 (socket says 172.21.15.204:50560) 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 -- 172.21.15.204:0/4051883333 learned_addr learned my addr 172.21.15.204:0/4051883333 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa820987700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 0x7fa81c19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 -- 172.21.15.204:0/4051883333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 msgr2=0x7fa81c19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 0x7fa81c19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 -- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa81c102c90 msgr2=0x7fa81c1a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa81c102c90 0x7fa81c1a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 -- 172.21.15.204:0/4051883333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1a23c0 con 0x7fa81c1010f0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa820987700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 0x7fa81c19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.045+0000 7fa821188700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00b530 tx=0x7fa80c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.046+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c015020 con 0x7fa81c1010f0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.046+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c011d10 con 0x7fa81c1010f0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.046+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c1a26b0 con 0x7fa81c1010f0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.046+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00f2e0 con 0x7fa81c1010f0 2024-03-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.046+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1a2c70 con 0x7fa81c1010f0 2024-03-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.047+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c068e50 con 0x7fa81c1010f0 2024-03-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.048+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa80c018020 con 0x7fa81c1010f0 2024-03-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.048+0000 7fa8127fc700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 0x7fa8080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.048+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa80c0cdd60 con 0x7fa81c1010f0 2024-03-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.049+0000 7fa820987700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 0x7fa8080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.050+0000 7fa820987700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 0x7fa8080a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa818007900 tx=0x7fa818008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.053+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa80c01f020 con 0x7fa81c1010f0 2024-03-24T00:34:48.260 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.260+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fa81c0fe8e0 con 0x7fa8080a2910 2024-03-24T00:34:48.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.278+0000 7fa8127fc700 1 -- 172.21.15.204:0/4051883333 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa81c0fe8e0 con 0x7fa8080a2910 2024-03-24T00:34:48.278 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mds.all update... 2024-03-24T00:34:48.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 msgr2=0x7fa8080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:48.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 0x7fa8080a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa818007900 tx=0x7fa818008040 comp rx=0 tx=0).stop 2024-03-24T00:34:48.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 msgr2=0x7fa81c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:48.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00b530 tx=0x7fa80c00b8f0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 shutdown_connections 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8080a2910 0x7fa8080a4dd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa81c102c90 0x7fa81c1a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81c1010f0 0x7fa81c19a760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 --2- 172.21.15.204:0/4051883333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa81c1022f0 0x7fa81c19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 >> 172.21.15.204:0/4051883333 conn(0x7fa81c0fc690 msgr2=0x7fa81c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 shutdown_connections 2024-03-24T00:34:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-24T00:34:48.280+0000 7fa8233ec700 1 -- 172.21.15.204:0/4051883333 wait complete. 2024-03-24T00:34:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:48 smithi177 ceph-mon[36034]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:48 smithi204 ceph-mon[36345]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:48 smithi123 ceph-mon[31487]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:48.952 DEBUG:teuthology.orchestra.run.smithi123:mds.a> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T00:34:48.958 DEBUG:teuthology.orchestra.run.smithi123:mds.d> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T00:34:49.004 DEBUG:teuthology.orchestra.run.smithi123:mds.g> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T00:34:49.050 DEBUG:teuthology.orchestra.run.smithi123:mds.j> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T00:34:49.095 DEBUG:teuthology.orchestra.run.smithi177:mds.b> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T00:34:49.098 DEBUG:teuthology.orchestra.run.smithi177:mds.e> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T00:34:49.141 DEBUG:teuthology.orchestra.run.smithi177:mds.h> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T00:34:49.184 DEBUG:teuthology.orchestra.run.smithi177:mds.k> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T00:34:49.227 DEBUG:teuthology.orchestra.run.smithi204:mds.c> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T00:34:49.229 DEBUG:teuthology.orchestra.run.smithi204:mds.f> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T00:34:49.273 DEBUG:teuthology.orchestra.run.smithi204:mds.i> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T00:34:49.316 DEBUG:teuthology.orchestra.run.smithi204:mds.l> sudo journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T00:34:49.359 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T00:34:49.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:49 smithi177 ceph-mon[36034]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:49 smithi177 ceph-mon[36034]: Saving service mds.all spec with placement smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-03-24T00:34:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:49 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:49 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:49.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:49 smithi204 ceph-mon[36345]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:49.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:49 smithi204 ceph-mon[36345]: Saving service mds.all spec with placement smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-03-24T00:34:49.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:49.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:49 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 -- 172.21.15.123:0/444843963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 msgr2=0x7f849c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 --2- 172.21.15.123:0/444843963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c104f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f848c009860 tx=0x7f848c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 -- 172.21.15.123:0/444843963 shutdown_connections 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 --2- 172.21.15.123:0/444843963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 --2- 172.21.15.123:0/444843963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c104f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 --2- 172.21.15.123:0/444843963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 -- 172.21.15.123:0/444843963 >> 172.21.15.123:0/444843963 conn(0x7f849c0fee50 msgr2=0x7f849c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 -- 172.21.15.123:0/444843963 shutdown_connections 2024-03-24T00:34:49.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.590+0000 7f84a4800640 1 -- 172.21.15.123:0/444843963 wait complete. 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.591+0000 7f84a4800640 1 Processor -- start 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.591+0000 7f84a4800640 1 -- start start 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c10f540 con 0x7f849c105490 2024-03-24T00:34:49.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c10f3c0 con 0x7f849c1038e0 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a4800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c10f6c0 con 0x7f849c104ad0 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:60772/0 (socket says 172.21.15.123:60772) 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1e00640 1 -- 172.21.15.123:0/1838335002 learned_addr learned my addr 172.21.15.123:0/1838335002 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46032/0 (socket says 172.21.15.123:46032) 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 -- 172.21.15.123:0/1838335002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 msgr2=0x7f849c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 -- 172.21.15.123:0/1838335002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 msgr2=0x7f849c199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a2800640 1 -- 172.21.15.123:0/1838335002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c1a1800 con 0x7f849c105490 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.592+0000 7f84a1400640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:49.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f84a2800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c1a1190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f849800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849800ce20 con 0x7f849c105490 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c1a1af0 con 0x7f849c105490 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8498005020 con 0x7f849c105490 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498010720 con 0x7f849c105490 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.593+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c1a2010 con 0x7f849c105490 2024-03-24T00:34:49.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.594+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c10ac80 con 0x7f849c105490 2024-03-24T00:34:49.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.596+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8498005540 con 0x7f849c105490 2024-03-24T00:34:49.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.597+0000 7f8496a00640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 0x7f84840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:49.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.597+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f84980ced40 con 0x7f849c105490 2024-03-24T00:34:49.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.598+0000 7f84a1e00640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 0x7f84840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:49.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.598+0000 7f84a1e00640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 0x7f84840a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8490009c60 tx=0x7f8490009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:49.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.602+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f849809c1e0 con 0x7f849c105490 2024-03-24T00:34:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:49 smithi123 ceph-mon[31487]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:34:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:49 smithi123 ceph-mon[31487]: Saving service mds.all spec with placement smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-03-24T00:34:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:49 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:49.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.801+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f849c0622a0 con 0x7f849c105490 2024-03-24T00:34:49.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.803+0000 7f8496a00640 1 -- 172.21.15.123:0/1838335002 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18515 (secure 0 0 0) 0x7f849809bae0 con 0x7f849c105490 2024-03-24T00:34:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:49.804 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:34:34.673592+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.805+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 msgr2=0x7f84840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 0x7f84840a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8490009c60 tx=0x7f8490009260 comp rx=0 tx=0).stop 2024-03-24T00:34:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 msgr2=0x7f849c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:49.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c1a1190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f849800ef10 comp rx=0 tx=0).stop 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 shutdown_connections 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f84840a2800 0x7f84840a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f849c105490 0x7f849c1a1190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f849c104ad0 0x7f849c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 --2- 172.21.15.123:0/1838335002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f849c1038e0 0x7f849c199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:49.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 >> 172.21.15.123:0/1838335002 conn(0x7f849c0fee50 msgr2=0x7f849c107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 shutdown_connections 2024-03-24T00:34:49.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:49.806+0000 7f84a4800640 1 -- 172.21.15.123:0/1838335002 wait complete. 2024-03-24T00:34:49.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:34:49.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1150583369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c54105470 msgr2=0x7f2c5410bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1150583369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c54105470 0x7f2c5410bd00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009be0 tx=0x7f2c4c02bef0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1150583369 shutdown_connections 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1150583369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c54105470 0x7f2c5410bd00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1150583369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c54104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1150583369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c541038c0 0x7f2c54103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1150583369 >> 172.21.15.123:0/1150583369 conn(0x7f2c540fee50 msgr2=0x7f2c54101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1150583369 shutdown_connections 2024-03-24T00:34:50.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.041+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1150583369 wait complete. 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 Processor -- start 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 -- start start 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c541038c0 0x7f2c54199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c54105470 0x7f2c541a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5410f520 con 0x7f2c54104ab0 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5410f3a0 con 0x7f2c54105470 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c5aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5410f6a0 con 0x7f2c541038c0 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c53400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c58a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c54105470 0x7f2c541a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c53400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48322/0 (socket says 172.21.15.123:48322) 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c53400640 1 -- 172.21.15.123:0/1052023219 learned_addr learned my addr 172.21.15.123:0/1052023219 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.042+0000 7f2c53e00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c541038c0 0x7f2c54199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 -- 172.21.15.123:0/1052023219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c541038c0 msgr2=0x7f2c54199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c541038c0 0x7f2c54199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 -- 172.21.15.123:0/1052023219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c54105470 msgr2=0x7f2c541a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c54105470 0x7f2c541a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 -- 172.21.15.123:0/1052023219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c541a1780 con 0x7f2c54104ab0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c53400640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000e9c0 tx=0x7f2c4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4000cde0 con 0x7f2c54104ab0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c400035f0 con 0x7f2c54104ab0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c541a1a70 con 0x7f2c54104ab0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.043+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40010690 con 0x7f2c54104ab0 2024-03-24T00:34:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.044+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c541a1fc0 con 0x7f2c54104ab0 2024-03-24T00:34:50.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.045+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5410ab80 con 0x7f2c54104ab0 2024-03-24T00:34:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.047+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2c400037a0 con 0x7f2c54104ab0 2024-03-24T00:34:50.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.049+0000 7f2c50c00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 0x7f2c380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.050+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2c400ce990 con 0x7f2c54104ab0 2024-03-24T00:34:50.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.050+0000 7f2c53e00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 0x7f2c380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.050+0000 7f2c53e00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 0x7f2c380a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c54104910 tx=0x7f2c44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:50.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.054+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c4009be30 con 0x7f2c54104ab0 2024-03-24T00:34:50.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.251+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c540622a0 con 0x7f2c54104ab0 2024-03-24T00:34:50.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.251+0000 7f2c50c00640 1 -- 172.21.15.123:0/1052023219 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c4009b730 con 0x7f2c54104ab0 2024-03-24T00:34:50.252 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 msgr2=0x7f2c380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 0x7f2c380a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c54104910 tx=0x7f2c44005f90 comp rx=0 tx=0).stop 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 msgr2=0x7f2c5419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000e9c0 tx=0x7f2c4000ee80 comp rx=0 tx=0).stop 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 shutdown_connections 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c54105470 0x7f2c541a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2c380a2800 0x7f2c380a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c54104ab0 0x7f2c5419a0a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 --2- 172.21.15.123:0/1052023219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c541038c0 0x7f2c54199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 >> 172.21.15.123:0/1052023219 conn(0x7f2c540fee50 msgr2=0x7f2c54107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 shutdown_connections 2024-03-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.254+0000 7f2c5aa00640 1 -- 172.21.15.123:0/1052023219 wait complete. 2024-03-24T00:34:50.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1838335002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:50.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:50 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1052023219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 -- 172.21.15.123:0/2195900197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220077c40 msgr2=0x7f92200784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 --2- 172.21.15.123:0/2195900197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220077c40 0x7f92200784e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9218009900 tx=0x7f921802bc30 comp rx=0 tx=0).stop 2024-03-24T00:34:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 -- 172.21.15.123:0/2195900197 shutdown_connections 2024-03-24T00:34:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 --2- 172.21.15.123:0/2195900197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220077c40 0x7f92200784e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 --2- 172.21.15.123:0/2195900197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f9220077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 --2- 172.21.15.123:0/2195900197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220076060 0x7f9220076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 -- 172.21.15.123:0/2195900197 >> 172.21.15.123:0/2195900197 conn(0x7f92200fee50 msgr2=0x7f9220101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:50.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 -- 172.21.15.123:0/2195900197 shutdown_connections 2024-03-24T00:34:50.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.483+0000 7f9228600640 1 -- 172.21.15.123:0/2195900197 wait complete. 2024-03-24T00:34:50.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.484+0000 7f9228600640 1 Processor -- start 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.484+0000 7f9228600640 1 -- start start 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f922019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 0x7f92201a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92201139b0 con 0x7f92200772a0 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220113830 con 0x7f9220077c40 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9228600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220113b30 con 0x7f9220076060 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46062/0 (socket says 172.21.15.123:46062) 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f922019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 -- 172.21.15.123:0/3426322986 learned_addr learned my addr 172.21.15.123:0/3426322986 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9226600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 0x7f92201a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 -- 172.21.15.123:0/3426322986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 msgr2=0x7f92201a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 0x7f92201a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.485+0000 7f9225c00640 1 -- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 msgr2=0x7f922019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9225c00640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f922019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9225c00640 1 -- 172.21.15.123:0/3426322986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92201a5b50 con 0x7f9220076060 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9226600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 0x7f92201a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9225200640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f922019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9225c00640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f921000e9f0 tx=0x7f921000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921000cd80 con 0x7f9220076060 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9210010910 con 0x7f9220076060 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210018990 con 0x7f9220076060 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92201a5e40 con 0x7f9220076060 2024-03-24T00:34:50.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.486+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92201a6250 con 0x7f9220076060 2024-03-24T00:34:50.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.487+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9220110360 con 0x7f9220076060 2024-03-24T00:34:50.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.489+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9210010ac0 con 0x7f9220076060 2024-03-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.490+0000 7f921e000640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 0x7f92040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.491+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92100ceb20 con 0x7f9220076060 2024-03-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.491+0000 7f9225200640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 0x7f92040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.491+0000 7f9225200640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 0x7f92040a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f922019f440 tx=0x7f920c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.493+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f921009bfc0 con 0x7f9220076060 2024-03-24T00:34:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1838335002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:50.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:50 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1052023219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1838335002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:34:50.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:50 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1052023219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:34:50.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.732+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f922002d3c0 con 0x7f9220076060 2024-03-24T00:34:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.733+0000 7f921e000640 1 -- 172.21.15.123:0/3426322986 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f922002d3c0 con 0x7f9220076060 2024-03-24T00:34:50.734 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:50.734 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T00:34:50.734 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 1 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.735+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 msgr2=0x7f92040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.735+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 0x7f92040a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f922019f440 tx=0x7f920c009500 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.735+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 msgr2=0x7f922019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.735+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f921000e9f0 tx=0x7f921000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 shutdown_connections 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9220077c40 0x7f92201a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f92040a2880 0x7f92040a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92200772a0 0x7f922019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 --2- 172.21.15.123:0/3426322986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9220076060 0x7f922019df30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 >> 172.21.15.123:0/3426322986 conn(0x7f92200fee50 msgr2=0x7f9220100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 shutdown_connections 2024-03-24T00:34:50.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:50.736+0000 7f9228600640 1 -- 172.21.15.123:0/3426322986 wait complete. 2024-03-24T00:34:51.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:51 smithi204 ceph-mon[36345]: Deploying daemon mds.l on smithi204 2024-03-24T00:34:51.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:51 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3426322986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:51.571 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 systemd[1]: Starting Ceph mds.l for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:34:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:51 smithi123 ceph-mon[31487]: Deploying daemon mds.l on smithi204 2024-03-24T00:34:51.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:51 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3426322986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:51 smithi177 ceph-mon[36034]: Deploying daemon mds.l on smithi204 2024-03-24T00:34:51.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:51 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3426322986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:51.947 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 podman[66875]: 2024-03-24 00:34:51.695473704 +0000 UTC m=+0.110797590 container create 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T00:34:51.948 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 podman[66875]: 2024-03-24 00:34:51.602278567 +0000 UTC m=+0.017602459 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:34:51.948 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 podman[66875]: 2024-03-24 00:34:51.919971489 +0000 UTC m=+0.335295391 container init 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T00:34:51.948 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 podman[66875]: 2024-03-24 00:34:51.922911836 +0000 UTC m=+0.338235726 container start 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:34:51.948 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 ceph-mds[66893]: -- 172.21.15.204:0/3340789967 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e96d27a40 con 0x564e96cfb000 2024-03-24T00:34:52.288 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l[66889]: starting mds.l at 2024-03-24T00:34:52.288 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:52 smithi204 bash[66875]: 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 2024-03-24T00:34:52.289 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:34:52 smithi204 systemd[1]: Started Ceph mds.l for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:34:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.204:6832/812925941,v1:172.21.15.204:6833/812925941] up:boot 2024-03-24T00:34:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: fsmap 1 up:standby 2024-03-24T00:34:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:34:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:52.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:52.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:52 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.204:6832/812925941,v1:172.21.15.204:6833/812925941] up:boot 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: fsmap 1 up:standby 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:52 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.204:6832/812925941,v1:172.21.15.204:6833/812925941] up:boot 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: fsmap 1 up:standby 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:52 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:52.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:52.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 -- 172.21.15.123:0/678755170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 msgr2=0x7fc748101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 --2- 172.21.15.123:0/678755170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748101930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc738009900 tx=0x7fc73802bc40 comp rx=0 tx=0).stop 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 -- 172.21.15.123:0/678755170 shutdown_connections 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 --2- 172.21.15.123:0/678755170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7481030c0 0x7fc748109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 --2- 172.21.15.123:0/678755170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc748102700 0x7fc748102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 --2- 172.21.15.123:0/678755170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748101930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 -- 172.21.15.123:0/678755170 >> 172.21.15.123:0/678755170 conn(0x7fc7480fca80 msgr2=0x7fc7480feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 -- 172.21.15.123:0/678755170 shutdown_connections 2024-03-24T00:34:52.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.979+0000 7fc74ce00640 1 -- 172.21.15.123:0/678755170 wait complete. 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.980+0000 7fc74ce00640 1 Processor -- start 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 -- start start 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 0x7fc748199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 0x7fc7481a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74810d1f0 con 0x7fc7481030c0 2024-03-24T00:34:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74810d070 con 0x7fc748101510 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc74ce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74810d370 con 0x7fc748102700 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc746a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 0x7fc748199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc747400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc747e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 0x7fc7481a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc746a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 0x7fc748199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46080/0 (socket says 172.21.15.123:46080) 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.981+0000 7fc746a00640 1 -- 172.21.15.123:0/3903317093 learned_addr learned my addr 172.21.15.123:0/3903317093 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 -- 172.21.15.123:0/3903317093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 msgr2=0x7fc748199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 0x7fc748199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 -- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 msgr2=0x7fc7481a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 0x7fc7481a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 -- 172.21.15.123:0/3903317093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7481a1620 con 0x7fc748101510 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747e00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 0x7fc7481a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc747400640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748199a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc738009230 tx=0x7fc738005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738041070 con 0x7fc748101510 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7380054e0 con 0x7fc748101510 2024-03-24T00:34:52.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7481a18b0 con 0x7fc748101510 2024-03-24T00:34:52.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738034440 con 0x7fc748101510 2024-03-24T00:34:52.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.982+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7481a1dc0 con 0x7fc748101510 2024-03-24T00:34:52.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.983+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7481a1fc0 con 0x7fc748101510 2024-03-24T00:34:52.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.985+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc738044070 con 0x7fc748101510 2024-03-24T00:34:52.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.988+0000 7fc73fe00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 0x7fc72c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:52.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.988+0000 7fc746a00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 0x7fc72c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:52.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.988+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc7380f00b0 con 0x7fc748101510 2024-03-24T00:34:52.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.989+0000 7fc746a00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 0x7fc72c0a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc7340095f0 tx=0x7fc734005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:52.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:52.991+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc7380bcf00 con 0x7fc748101510 2024-03-24T00:34:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.232+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7481a2190 con 0x7fc748101510 2024-03-24T00:34:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.233+0000 7fc73fe00640 1 -- 172.21.15.123:0/3903317093 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7fc7380c0fa0 con 0x7fc748101510 2024-03-24T00:34:53.235 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:53.235 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:34:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 2 2024-03-24T00:34:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 msgr2=0x7fc72c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 0x7fc72c0a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc7340095f0 tx=0x7fc734005f90 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 msgr2=0x7fc748199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748199a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc738009230 tx=0x7fc738005d50 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 shutdown_connections 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc72c0a2880 0x7fc72c0a4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7481030c0 0x7fc7481a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc748102700 0x7fc748199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 --2- 172.21.15.123:0/3903317093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc748101510 0x7fc748199a00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.236+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 >> 172.21.15.123:0/3903317093 conn(0x7fc7480fca80 msgr2=0x7fc7480738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.237+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 shutdown_connections 2024-03-24T00:34:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:53.237+0000 7fc74ce00640 1 -- 172.21.15.123:0/3903317093 wait complete. 2024-03-24T00:34:53.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:53 smithi177 ceph-mon[36034]: Deploying daemon mds.k on smithi177 2024-03-24T00:34:53.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:53 smithi177 ceph-mon[36034]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:53.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3903317093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:53.536 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:53 smithi123 ceph-mon[31487]: Deploying daemon mds.k on smithi177 2024-03-24T00:34:53.536 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:53 smithi123 ceph-mon[31487]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:53.536 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3903317093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:53 smithi204 ceph-mon[36345]: Deploying daemon mds.k on smithi177 2024-03-24T00:34:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:53 smithi204 ceph-mon[36345]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3903317093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:54.462 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 systemd[1]: Starting Ceph mds.k for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:34:54.845 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 podman[66049]: 2024-03-24 00:34:54.481582743 +0000 UTC m=+0.017992873 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:34:54.845 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 podman[66049]: 2024-03-24 00:34:54.583460682 +0000 UTC m=+0.119870792 container create 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T00:34:54.845 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 podman[66049]: 2024-03-24 00:34:54.819554824 +0000 UTC m=+0.355964936 container init 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:34:54.845 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 podman[66049]: 2024-03-24 00:34:54.822389738 +0000 UTC m=+0.358799855 container start 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True) 2024-03-24T00:34:55.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 ceph-mds[66068]: -- 172.21.15.177:0/2037125367 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562f4a3dfa40 con 0x562f4a3b2c00 2024-03-24T00:34:55.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k[66064]: starting mds.k at 2024-03-24T00:34:55.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 bash[66049]: 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea 2024-03-24T00:34:55.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:34:54 smithi177 systemd[1]: Started Ceph mds.k for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:34:55.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 -- 172.21.15.123:0/3431548998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c077c40 msgr2=0x7fb22c0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 --2- 172.21.15.123:0/3431548998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c077c40 0x7fb22c0784e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb224009ca0 tx=0x7fb22402c100 comp rx=0 tx=0).stop 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 -- 172.21.15.123:0/3431548998 shutdown_connections 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 --2- 172.21.15.123:0/3431548998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c077c40 0x7fb22c0784e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 --2- 172.21.15.123:0/3431548998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c0772a0 0x7fb22c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 --2- 172.21.15.123:0/3431548998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c076060 0x7fb22c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 -- 172.21.15.123:0/3431548998 >> 172.21.15.123:0/3431548998 conn(0x7fb22c0fee90 msgr2=0x7fb22c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:55.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 -- 172.21.15.123:0/3431548998 shutdown_connections 2024-03-24T00:34:55.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.468+0000 7fb231200640 1 -- 172.21.15.123:0/3431548998 wait complete. 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 Processor -- start 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 -- start start 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c076060 0x7fb22c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c113970 con 0x7fb22c0772a0 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c1137f0 con 0x7fb22c077c40 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.469+0000 7fb231200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c113af0 con 0x7fb22c076060 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:55.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48380/0 (socket says 172.21.15.123:48380) 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 -- 172.21.15.123:0/4139819336 learned_addr learned my addr 172.21.15.123:0/4139819336 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229e00640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c076060 0x7fb22c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb22a800640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 -- 172.21.15.123:0/4139819336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c076060 msgr2=0x7fb22c19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c076060 0x7fb22c19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 -- 172.21.15.123:0/4139819336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c077c40 msgr2=0x7fb22c1a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 -- 172.21.15.123:0/4139819336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c1a5be0 con 0x7fb22c0772a0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.470+0000 7fb229400640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e9f0 tx=0x7fb21400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.471+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21400cd80 con 0x7fb22c0772a0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.471+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb214005500 con 0x7fb22c0772a0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.471+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c1a5ed0 con 0x7fb22c0772a0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.471+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214010690 con 0x7fb22c0772a0 2024-03-24T00:34:55.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.471+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c1a6340 con 0x7fb22c0772a0 2024-03-24T00:34:55.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.472+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c1103e0 con 0x7fb22c0772a0 2024-03-24T00:34:55.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.475+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb2140046c0 con 0x7fb22c0772a0 2024-03-24T00:34:55.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.477+0000 7fb222a00640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 0x7fb2100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:55.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.477+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb2140cea30 con 0x7fb22c0772a0 2024-03-24T00:34:55.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.477+0000 7fb229e00640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 0x7fb2100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:55.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.478+0000 7fb229e00640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 0x7fb2100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009bd0 tx=0x7fb21c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:55.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.481+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb21409bed0 con 0x7fb22c0772a0 2024-03-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.723+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb22c0622a0 con 0x7fb22c0772a0 2024-03-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.724+0000 7fb222a00640 1 -- 172.21.15.123:0/4139819336 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fb21409b7d0 con 0x7fb22c0772a0 2024-03-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 3 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 msgr2=0x7fb2100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 0x7fb2100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009bd0 tx=0x7fb21c005f90 comp rx=0 tx=0).stop 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 msgr2=0x7fb22c19e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e9f0 tx=0x7fb21400edf0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 shutdown_connections 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c077c40 0x7fb22c1a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb2100a2800 0x7fb2100a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb22c0772a0 0x7fb22c19e500 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 --2- 172.21.15.123:0/4139819336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb22c076060 0x7fb22c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.726+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 >> 172.21.15.123:0/4139819336 conn(0x7fb22c0fee90 msgr2=0x7fb22c100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.727+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 shutdown_connections 2024-03-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:55.727+0000 7fb231200640 1 -- 172.21.15.123:0/4139819336 wait complete. 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6832/3661966001,v1:172.21.15.177:6833/3661966001] up:boot 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: fsmap 2 up:standby 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: Deploying daemon mds.j on smithi123 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:56.001 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4139819336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6832/3661966001,v1:172.21.15.177:6833/3661966001] up:boot 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: fsmap 2 up:standby 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: Deploying daemon mds.j on smithi123 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4139819336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:56.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6832/3661966001,v1:172.21.15.177:6833/3661966001] up:boot 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: fsmap 2 up:standby 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: Deploying daemon mds.j on smithi123 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:56.275 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4139819336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:57.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 -- 172.21.15.123:0/2430094146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32281038e0 msgr2=0x7f3228103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 --2- 172.21.15.123:0/2430094146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32281038e0 0x7f3228103d00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f321c009ca0 tx=0x7f321c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 -- 172.21.15.123:0/2430094146 shutdown_connections 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 --2- 172.21.15.123:0/2430094146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3228105490 0x7f322810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 --2- 172.21.15.123:0/2430094146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228104ad0 0x7f3228104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 --2- 172.21.15.123:0/2430094146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32281038e0 0x7f3228103d00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 -- 172.21.15.123:0/2430094146 >> 172.21.15.123:0/2430094146 conn(0x7f32280fee90 msgr2=0x7f32281012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 -- 172.21.15.123:0/2430094146 shutdown_connections 2024-03-24T00:34:58.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.078+0000 7f322fe00640 1 -- 172.21.15.123:0/2430094146 wait complete. 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 Processor -- start 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 -- start start 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32281038e0 0x7f3228199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228105490 0x7f32281a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322810f530 con 0x7f3228105490 2024-03-24T00:34:58.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322810f3b0 con 0x7f3228104ad0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.079+0000 7f322fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322810f6b0 con 0x7f32281038e0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37120/0 (socket says 172.21.15.123:37120) 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 -- 172.21.15.123:0/2051845474 learned_addr learned my addr 172.21.15.123:0/2051845474 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322de00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228105490 0x7f32281a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322d400640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32281038e0 0x7f3228199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 -- 172.21.15.123:0/2051845474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32281038e0 msgr2=0x7f3228199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32281038e0 0x7f3228199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 -- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228105490 msgr2=0x7f32281a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228105490 0x7f32281a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 -- 172.21.15.123:0/2051845474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32281a1830 con 0x7f3228104ad0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.080+0000 7f322ca00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.081+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400cd80 con 0x7f3228104ad0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.081+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3214010910 con 0x7f3228104ad0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.081+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32281a1b20 con 0x7f3228104ad0 2024-03-24T00:34:58.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.081+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214018990 con 0x7f3228104ad0 2024-03-24T00:34:58.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.081+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32281a2040 con 0x7f3228104ad0 2024-03-24T00:34:58.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.082+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322810ac80 con 0x7f3228104ad0 2024-03-24T00:34:58.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.085+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3214010ac0 con 0x7f3228104ad0 2024-03-24T00:34:58.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.086+0000 7f3226000640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 0x7f32100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.086+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f32140ce9c0 con 0x7f3228104ad0 2024-03-24T00:34:58.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.086+0000 7f322d400640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 0x7f32100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.087+0000 7f322d400640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 0x7f32100a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f321c02c630 tx=0x7f321c009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.090+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f321409b890 con 0x7f3228104ad0 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.336+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f322802d270 con 0x7f3228104ad0 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.337+0000 7f3226000640 1 -- 172.21.15.123:0/2051845474 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f321409f930 con 0x7f3228104ad0 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:34:58.339 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 3 2024-03-24T00:34:58.340 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:58 smithi123 ceph-mon[31487]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 msgr2=0x7f32100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 0x7f32100a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f321c02c630 tx=0x7f321c009360 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 msgr2=0x7f322819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 shutdown_connections 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32100a2800 0x7f32100a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3228105490 0x7f32281a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3228104ad0 0x7f322819a160 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 --2- 172.21.15.123:0/2051845474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32281038e0 0x7f3228199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 >> 172.21.15.123:0/2051845474 conn(0x7f32280fee90 msgr2=0x7f3228107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 shutdown_connections 2024-03-24T00:34:58.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:34:58.340+0000 7f322fe00640 1 -- 172.21.15.123:0/2051845474 wait complete. 2024-03-24T00:34:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:58 smithi177 ceph-mon[36034]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:58 smithi204 ceph-mon[36345]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:34:58.608 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:34:58 smithi123 podman[71775]: 2024-03-24 00:34:58.584192988 +0000 UTC m=+0.370156089 container start 1658ba157df08b20b492d0638ebb258c2268bc736ffaf714925221389a44395e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T00:34:59.106 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:34:58 smithi123 ceph-mds[71796]: -- 172.21.15.123:0/1156958206 <== mon.0 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c19679a40 con 0x558c1964d000 2024-03-24T00:34:59.106 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:34:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j[71792]: starting mds.j at 2024-03-24T00:34:59.107 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:34:58 smithi123 bash[71775]: 1658ba157df08b20b492d0638ebb258c2268bc736ffaf714925221389a44395e 2024-03-24T00:34:59.107 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:34:58 smithi123 systemd[1]: Started Ceph mds.j for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:34:59.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2051845474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:59.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:59.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:34:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2051845474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:34:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2051845474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:34:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:34:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:34:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:34:59.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:34:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:00.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:00.448 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:00 smithi123 ceph-mon[31487]: Deploying daemon mds.d on smithi123 2024-03-24T00:35:00.448 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:00 smithi123 ceph-mon[31487]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:00.448 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:00 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.123:6834/548064251,v1:172.21.15.123:6835/548064251] up:boot 2024-03-24T00:35:00.448 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:00 smithi123 ceph-mon[31487]: fsmap 3 up:standby 2024-03-24T00:35:00.448 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:00 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:00 smithi177 ceph-mon[36034]: Deploying daemon mds.d on smithi123 2024-03-24T00:35:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:00 smithi177 ceph-mon[36034]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:00.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:00 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.123:6834/548064251,v1:172.21.15.123:6835/548064251] up:boot 2024-03-24T00:35:00.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:00 smithi177 ceph-mon[36034]: fsmap 3 up:standby 2024-03-24T00:35:00.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:00 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:00 smithi204 ceph-mon[36345]: Deploying daemon mds.d on smithi123 2024-03-24T00:35:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:00 smithi204 ceph-mon[36345]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:00 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.123:6834/548064251,v1:172.21.15.123:6835/548064251] up:boot 2024-03-24T00:35:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:00 smithi204 ceph-mon[36345]: fsmap 3 up:standby 2024-03-24T00:35:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:00 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- 172.21.15.123:0/2223162247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81036f0 msgr2=0x7fe3d8103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 --2- 172.21.15.123:0/2223162247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81036f0 0x7fe3d8103b10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009900 tx=0x7fe3c802bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- 172.21.15.123:0/2223162247 shutdown_connections 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 --2- 172.21.15.123:0/2223162247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 0x7fe3d810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 --2- 172.21.15.123:0/2223162247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81048e0 0x7fe3d8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 --2- 172.21.15.123:0/2223162247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81036f0 0x7fe3d8103b10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- 172.21.15.123:0/2223162247 >> 172.21.15.123:0/2223162247 conn(0x7fe3d80fec80 msgr2=0x7fe3d81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- 172.21.15.123:0/2223162247 shutdown_connections 2024-03-24T00:35:00.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- 172.21.15.123:0/2223162247 wait complete. 2024-03-24T00:35:00.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 Processor -- start 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.585+0000 7fe3dd600640 1 -- start start 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 0x7fe3d81a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f3b0 con 0x7fe3d81048e0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f230 con 0x7fe3d81052a0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f530 con 0x7fe3d81036f0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39256/0 (socket says 172.21.15.123:39256) 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7400640 1 -- 172.21.15.123:0/3440675129 learned_addr learned my addr 172.21.15.123:0/3440675129 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:59628/0 (socket says 172.21.15.123:59628) 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dcc00640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 0x7fe3d81a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 -- 172.21.15.123:0/3440675129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 msgr2=0x7fe3d81a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 0x7fe3d81a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 -- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 msgr2=0x7fe3d8199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 -- 172.21.15.123:0/3440675129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d81a1680 con 0x7fe3d81036f0 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7400640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d7e00640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009230 tx=0x7fe3c802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8043020 con 0x7fe3d81036f0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3c80054e0 con 0x7fe3d81036f0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d81a1910 con 0x7fe3d81036f0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8003af0 con 0x7fe3d81036f0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.590+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d81a1e30 con 0x7fe3d81036f0 2024-03-24T00:35:00.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d810aa30 con 0x7fe3d81036f0 2024-03-24T00:35:00.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe3c8046020 con 0x7fe3d81036f0 2024-03-24T00:35:00.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3d4c00640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3c80f05f0 con 0x7fe3d81036f0 2024-03-24T00:35:00.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3d7400640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.593+0000 7fe3d7400640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe3d819af70 tx=0x7fe3c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.620+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe3c804d020 con 0x7fe3d81036f0 2024-03-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.870+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d80622a0 con 0x7fe3d81036f0 2024-03-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.871+0000 7fe3d4c00640 1 -- 172.21.15.123:0/3440675129 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2910 (secure 0 0 0) 0x7fe3c80bda90 con 0x7fe3d81036f0 2024-03-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4}],"filesystems":[]} 2024-03-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 4 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.873+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 msgr2=0x7fe3bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.873+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe3d819af70 tx=0x7fe3c4005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 msgr2=0x7fe3d8199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009230 tx=0x7fe3c802c5d0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 shutdown_connections 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d81052a0 0x7fe3d81a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d81048e0 0x7fe3d8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 --2- 172.21.15.123:0/3440675129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3d81036f0 0x7fe3d8199a60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 >> 172.21.15.123:0/3440675129 conn(0x7fe3d80fec80 msgr2=0x7fe3d8107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 shutdown_connections 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:00.874+0000 7fe3dd600640 1 -- 172.21.15.123:0/3440675129 wait complete. 2024-03-24T00:35:01.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:01 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3440675129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:01.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:01 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3440675129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:01 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3440675129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:02.071 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:01 smithi123 systemd[1]: Starting Ceph mds.d for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:02.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:02 smithi123 ceph-mon[31487]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:02.357 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 podman[72121]: 2024-03-24 00:35:01.975110474 +0000 UTC m=+0.032986312 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:02.357 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 podman[72121]: 2024-03-24 00:35:02.097619384 +0000 UTC m=+0.155495253 container create 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T00:35:02.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:02 smithi177 ceph-mon[36034]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:02 smithi204 ceph-mon[36345]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 podman[72121]: 2024-03-24 00:35:02.402944874 +0000 UTC m=+0.460820746 container init 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 podman[72121]: 2024-03-24 00:35:02.413366809 +0000 UTC m=+0.471242641 container start 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 ceph-mds[72140]: -- 172.21.15.123:0/1809609946 <== mon.0 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eeb0ac5a40 con 0x55eeb0a98c00 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d[72136]: starting mds.d at 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 bash[72121]: 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a 2024-03-24T00:35:02.857 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:35:02 smithi123 systemd[1]: Started Ceph mds.d for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:02.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 -- 172.21.15.123:0/1225655831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81016c0 msgr2=0x7fbeb8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 --2- 172.21.15.123:0/1225655831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81016c0 0x7fbeb8101ae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009860 tx=0x7fbea802bb70 comp rx=0 tx=0).stop 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 -- 172.21.15.123:0/1225655831 shutdown_connections 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 --2- 172.21.15.123:0/1225655831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb8103270 0x7fbeb8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 --2- 172.21.15.123:0/1225655831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb81028b0 0x7fbeb8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 --2- 172.21.15.123:0/1225655831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81016c0 0x7fbeb8101ae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 -- 172.21.15.123:0/1225655831 >> 172.21.15.123:0/1225655831 conn(0x7fbeb80fcc50 msgr2=0x7fbeb80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 -- 172.21.15.123:0/1225655831 shutdown_connections 2024-03-24T00:35:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.119+0000 7fbebee00640 1 -- 172.21.15.123:0/1225655831 wait complete. 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 Processor -- start 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 -- start start 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb81016c0 0x7fbeb81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 0x7fbeb819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb810d340 con 0x7fbeb81028b0 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb810d1c0 con 0x7fbeb8103270 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.120+0000 7fbebee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb810d4c0 con 0x7fbeb81016c0 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39290/0 (socket says 172.21.15.123:39290) 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 -- 172.21.15.123:0/3230892777 learned_addr learned my addr 172.21.15.123:0/3230892777 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbebce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 0x7fbeb819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7e00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb81016c0 0x7fbeb81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 -- 172.21.15.123:0/3230892777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb81016c0 msgr2=0x7fbeb81979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb81016c0 0x7fbeb81979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 -- 172.21.15.123:0/3230892777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 msgr2=0x7fbeb819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 0x7fbeb819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 -- 172.21.15.123:0/3230892777 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb819f5d0 con 0x7fbeb81028b0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbebce00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 0x7fbeb819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.121+0000 7fbeb7400640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbea000e9c0 tx=0x7fbea000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.122+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea000ccc0 con 0x7fbeb81028b0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.122+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbea0005500 con 0x7fbeb81028b0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.122+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb819f8c0 con 0x7fbeb81028b0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.122+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0010430 con 0x7fbeb81028b0 2024-03-24T00:35:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.122+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb819fe60 con 0x7fbeb81028b0 2024-03-24T00:35:03.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.123+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb81089e0 con 0x7fbeb81028b0 2024-03-24T00:35:03.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.125+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbea00046c0 con 0x7fbeb81028b0 2024-03-24T00:35:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.126+0000 7fbeb4c00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 0x7fbe9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.126+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbea00ce8d0 con 0x7fbeb81028b0 2024-03-24T00:35:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.127+0000 7fbeb7e00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 0x7fbe9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.127+0000 7fbeb7e00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 0x7fbe9c0a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbea80349e0 tx=0x7fbea8034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:03.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.131+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbea009bd70 con 0x7fbeb81028b0 2024-03-24T00:35:03.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.373+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeb802d070 con 0x7fbeb81028b0 2024-03-24T00:35:03.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.374+0000 7fbeb4c00640 1 -- 172.21.15.123:0/3230892777 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7fbea009b670 con 0x7fbeb81028b0 2024-03-24T00:35:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:03.376 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4}],"filesystems":[]} 2024-03-24T00:35:03.376 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 5 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.376+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 msgr2=0x7fbe9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.376+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 0x7fbe9c0a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbea80349e0 tx=0x7fbea8034ab0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.376+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 msgr2=0x7fbeb8197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.376+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbea000e9c0 tx=0x7fbea000ee80 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 shutdown_connections 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbeb8103270 0x7fbeb819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe9c0a2880 0x7fbe9c0a4d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbeb81028b0 0x7fbeb8197ef0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 --2- 172.21.15.123:0/3230892777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeb81016c0 0x7fbeb81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 >> 172.21.15.123:0/3230892777 conn(0x7fbeb80fcc50 msgr2=0x7fbeb8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 shutdown_connections 2024-03-24T00:35:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:03.377+0000 7fbebee00640 1 -- 172.21.15.123:0/3230892777 wait complete. 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: Deploying daemon mds.e on smithi177 2024-03-24T00:35:03.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:03.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.123:6836/2207292600,v1:172.21.15.123:6837/2207292600] up:boot 2024-03-24T00:35:03.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: fsmap 4 up:standby 2024-03-24T00:35:03.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:35:03.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:03 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3230892777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:04.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: Deploying daemon mds.e on smithi177 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.123:6836/2207292600,v1:172.21.15.123:6837/2207292600] up:boot 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: fsmap 4 up:standby 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:35:04.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:03 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3230892777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: Deploying daemon mds.e on smithi177 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.123:6836/2207292600,v1:172.21.15.123:6837/2207292600] up:boot 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: fsmap 4 up:standby 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:35:04.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:03 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3230892777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:04.962 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:04 smithi177 systemd[1]: Starting Ceph mds.e for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:05.365 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 podman[66369]: 2024-03-24 00:35:04.997265716 +0000 UTC m=+0.018142794 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:05.365 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 podman[66369]: 2024-03-24 00:35:05.112580697 +0000 UTC m=+0.133457749 container create e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:35:05.365 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 podman[66369]: 2024-03-24 00:35:05.365267515 +0000 UTC m=+0.386144579 container init e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T00:35:05.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:05.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 -- 172.21.15.123:0/1466482872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 msgr2=0x7f5894101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 --2- 172.21.15.123:0/1466482872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894101ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f588c009860 tx=0x7f588c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 -- 172.21.15.123:0/1466482872 shutdown_connections 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 --2- 172.21.15.123:0/1466482872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5894103270 0x7f5894109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 --2- 172.21.15.123:0/1466482872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58941028b0 0x7f5894102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 --2- 172.21.15.123:0/1466482872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894101ae0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.621+0000 7f589c800640 1 -- 172.21.15.123:0/1466482872 >> 172.21.15.123:0/1466482872 conn(0x7f58940fcc50 msgr2=0x7f58940ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.622+0000 7f589c800640 1 -- 172.21.15.123:0/1466482872 shutdown_connections 2024-03-24T00:35:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.622+0000 7f589c800640 1 -- 172.21.15.123:0/1466482872 wait complete. 2024-03-24T00:35:05.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.622+0000 7f589c800640 1 Processor -- start 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 -- start start 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 0x7f5894073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 0x7f5894073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410d340 con 0x7f5894103270 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410d1c0 con 0x7f58941016c0 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589c800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410d4c0 con 0x7f58941028b0 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f5899e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f5899400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 0x7f5894073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f5899400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 0x7f5894073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:59674/0 (socket says 172.21.15.123:59674) 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f5899e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37220/0 (socket says 172.21.15.123:37220) 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f5899400640 1 -- 172.21.15.123:0/3606482165 learned_addr learned my addr 172.21.15.123:0/3606482165 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.623+0000 7f589a800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 0x7f5894073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 -- 172.21.15.123:0/3606482165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 msgr2=0x7f5894073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 0x7f5894073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 -- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 msgr2=0x7f5894073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 0x7f5894073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 -- 172.21.15.123:0/3606482165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5894074860 con 0x7f58941016c0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f589a800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 0x7f5894073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f5899e00640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f588c009780 tx=0x7f588c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c041020 con 0x7f58941016c0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f588c02c9f0 con 0x7f58941016c0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58941a1aa0 con 0x7f58941016c0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c039040 con 0x7f58941016c0 2024-03-24T00:35:05.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.624+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941a2010 con 0x7f58941016c0 2024-03-24T00:35:05.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.625+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58941089e0 con 0x7f58941016c0 2024-03-24T00:35:05.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.627+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f588c044020 con 0x7f58941016c0 2024-03-24T00:35:05.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.628+0000 7f588aa00640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 0x7f58800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.629+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f588c0efe30 con 0x7f58941016c0 2024-03-24T00:35:05.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.629+0000 7f5899400640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 0x7f58800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.629+0000 7f5899400640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 0x7f58800a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58940751c0 tx=0x7f5884009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.633+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f588c04b020 con 0x7f58941016c0 2024-03-24T00:35:05.712 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 podman[66369]: 2024-03-24 00:35:05.368040119 +0000 UTC m=+0.388917177 container start e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:35:05.712 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 ceph-mds[66388]: -- 172.21.15.177:0/3524209959 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586b4edba40 con 0x5586b4eaf000 2024-03-24T00:35:05.712 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e[66384]: starting mds.e at 2024-03-24T00:35:05.713 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 bash[66369]: e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 2024-03-24T00:35:05.713 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:35:05 smithi177 systemd[1]: Started Ceph mds.e for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:05.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.872+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f589402d070 con 0x7f58941016c0 2024-03-24T00:35:05.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.873+0000 7f588aa00640 1 -- 172.21.15.123:0/3606482165 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7f588c0bd2d0 con 0x7f58941016c0 2024-03-24T00:35:05.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:05.874 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4}],"filesystems":[]} 2024-03-24T00:35:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 5 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.875+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 msgr2=0x7f58800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.875+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 0x7f58800a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58940751c0 tx=0x7f5884009260 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.875+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 msgr2=0x7f5894075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.875+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f588c009780 tx=0x7f588c032430 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.875+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 shutdown_connections 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f58800a2880 0x7f58800a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5894103270 0x7f5894073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58941028b0 0x7f5894073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 --2- 172.21.15.123:0/3606482165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58941016c0 0x7f5894075450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 >> 172.21.15.123:0/3606482165 conn(0x7f58940fcc50 msgr2=0x7f5894105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 shutdown_connections 2024-03-24T00:35:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:05.876+0000 7f589c800640 1 -- 172.21.15.123:0/3606482165 wait complete. 2024-03-24T00:35:06.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:06.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:06.358 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:06 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3606482165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:06.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:06.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:06.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:06.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:06.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:06 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3606482165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:06.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:06.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:06 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3606482165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:07.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:07 smithi177 ceph-mon[36034]: Deploying daemon mds.f on smithi204 2024-03-24T00:35:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:07 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6834/1091560059,v1:172.21.15.177:6835/1091560059] up:boot 2024-03-24T00:35:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:07 smithi177 ceph-mon[36034]: fsmap 5 up:standby 2024-03-24T00:35:07.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:35:07.484 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:07 smithi204 ceph-mon[36345]: Deploying daemon mds.f on smithi204 2024-03-24T00:35:07.485 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:07 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6834/1091560059,v1:172.21.15.177:6835/1091560059] up:boot 2024-03-24T00:35:07.485 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:07 smithi204 ceph-mon[36345]: fsmap 5 up:standby 2024-03-24T00:35:07.485 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:35:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:07 smithi123 ceph-mon[31487]: Deploying daemon mds.f on smithi204 2024-03-24T00:35:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:07 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6834/1091560059,v1:172.21.15.177:6835/1091560059] up:boot 2024-03-24T00:35:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:07 smithi123 ceph-mon[31487]: fsmap 5 up:standby 2024-03-24T00:35:07.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:35:07.821 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:07 smithi204 systemd[1]: Starting Ceph mds.f for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:07.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 -- 172.21.15.123:0/2842536361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101052a0 msgr2=0x7fe61010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 --2- 172.21.15.123:0/2842536361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101052a0 0x7fe61010bb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe608009ca0 tx=0x7fe60802c100 comp rx=0 tx=0).stop 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 -- 172.21.15.123:0/2842536361 shutdown_connections 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 --2- 172.21.15.123:0/2842536361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101052a0 0x7fe61010bb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 --2- 172.21.15.123:0/2842536361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 --2- 172.21.15.123:0/2842536361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101036f0 0x7fe610103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.116+0000 7fe615c00640 1 -- 172.21.15.123:0/2842536361 >> 172.21.15.123:0/2842536361 conn(0x7fe6100fec80 msgr2=0x7fe6101010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:08.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.117+0000 7fe615c00640 1 -- 172.21.15.123:0/2842536361 shutdown_connections 2024-03-24T00:35:08.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.117+0000 7fe615c00640 1 -- 172.21.15.123:0/2842536361 wait complete. 2024-03-24T00:35:08.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 Processor -- start 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 -- start start 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 0x7fe6101a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61010f370 con 0x7fe6101048e0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61010f1f0 con 0x7fe6101052a0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61010f4f0 con 0x7fe6101036f0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe60fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe607400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe60fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58798/0 (socket says 172.21.15.123:58798) 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe60fe00640 1 -- 172.21.15.123:0/748288665 learned_addr learned my addr 172.21.15.123:0/748288665 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.118+0000 7fe615200640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 0x7fe6101a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 -- 172.21.15.123:0/748288665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 msgr2=0x7fe6101a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 0x7fe6101a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 -- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 msgr2=0x7fe610199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 -- 172.21.15.123:0/748288665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6101a15f0 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe615200640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 0x7fe6101a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe607400640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60fe00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe60000e9c0 tx=0x7fe60000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000cd80 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6000035f0 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6101a18e0 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.119+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600010690 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.120+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6101a1de0 con 0x7fe6101036f0 2024-03-24T00:35:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.121+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61010a9b0 con 0x7fe6101036f0 2024-03-24T00:35:08.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.122+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe6000037a0 con 0x7fe6101036f0 2024-03-24T00:35:08.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.124+0000 7fe60d600640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.124+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe6000cea50 con 0x7fe6101036f0 2024-03-24T00:35:08.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.124+0000 7fe607400640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.124+0000 7fe607400640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe61019aee0 tx=0x7fe5f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:08.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.126+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe60009bef0 con 0x7fe6101036f0 2024-03-24T00:35:08.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:08 smithi204 ceph-mon[36345]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:08.150 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:07 smithi204 podman[67182]: 2024-03-24 00:35:07.789035901 +0000 UTC m=+0.018057484 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:08.150 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:07 smithi204 podman[67182]: 2024-03-24 00:35:07.915182844 +0000 UTC m=+0.144204413 container create e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T00:35:08.150 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 podman[67182]: 2024-03-24 00:35:08.150963593 +0000 UTC m=+0.379985180 container init e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:35:08.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:08 smithi123 ceph-mon[31487]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:08.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.370+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe61002d070 con 0x7fe6101036f0 2024-03-24T00:35:08.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.371+0000 7fe60d600640 1 -- 172.21.15.123:0/748288665 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4498 (secure 0 0 0) 0x7fe60009b7f0 con 0x7fe6101036f0 2024-03-24T00:35:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:08.372 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-24T00:35:08.372 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 6 2024-03-24T00:35:08.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.373+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 msgr2=0x7fe5f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.373+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe61019aee0 tx=0x7fe5f8005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.373+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 msgr2=0x7fe6101999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.373+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe60000e9c0 tx=0x7fe60000ee80 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 shutdown_connections 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6101052a0 0x7fe6101a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6101048e0 0x7fe610199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 --2- 172.21.15.123:0/748288665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6101036f0 0x7fe6101999d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 >> 172.21.15.123:0/748288665 conn(0x7fe6100fec80 msgr2=0x7fe610107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 shutdown_connections 2024-03-24T00:35:08.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:08.374+0000 7fe615c00640 1 -- 172.21.15.123:0/748288665 wait complete. 2024-03-24T00:35:08.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:08 smithi177 ceph-mon[36034]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:08.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 podman[67182]: 2024-03-24 00:35:08.153705149 +0000 UTC m=+0.382726725 container start e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:35:08.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 ceph-mds[67201]: -- 172.21.15.204:0/2280044777 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571f7c77a40 con 0x5571f7c4b000 2024-03-24T00:35:08.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f[67197]: starting mds.f at 2024-03-24T00:35:08.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 bash[67182]: e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 2024-03-24T00:35:08.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:35:08 smithi204 systemd[1]: Started Ceph mds.f for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/748288665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.204:6834/1788804579,v1:172.21.15.204:6835/1788804579] up:boot 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: fsmap 6 up:standby 2024-03-24T00:35:09.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:35:09.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/748288665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.204:6834/1788804579,v1:172.21.15.204:6835/1788804579] up:boot 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: fsmap 6 up:standby 2024-03-24T00:35:09.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:35:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/748288665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.204:6834/1788804579,v1:172.21.15.204:6835/1788804579] up:boot 2024-03-24T00:35:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: fsmap 6 up:standby 2024-03-24T00:35:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:35:10.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:10 smithi204 ceph-mon[36345]: Deploying daemon mds.c on smithi204 2024-03-24T00:35:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:10 smithi204 ceph-mon[36345]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:10.572 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 systemd[1]: Starting Ceph mds.c for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:10 smithi123 ceph-mon[31487]: Deploying daemon mds.c on smithi204 2024-03-24T00:35:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:10 smithi123 ceph-mon[31487]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.615+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1631131164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 msgr2=0x7f3fdc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.615+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1631131164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc104530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009900 tx=0x7f3fcc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1631131164 shutdown_connections 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1631131164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 0x7f3fdc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1631131164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1631131164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc104530 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1631131164 >> 172.21.15.123:0/1631131164 conn(0x7f3fdc0fbb30 msgr2=0x7f3fdc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:10.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1631131164 shutdown_connections 2024-03-24T00:35:10.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.616+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1631131164 wait complete. 2024-03-24T00:35:10.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 Processor -- start 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 -- start start 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 0x7f3fdc100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc10d3b0 con 0x7f3fdc104a70 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc10d230 con 0x7f3fdc1073a0 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fe0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc10d530 con 0x7f3fdc102140 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fd9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fd9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fd9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58816/0 (socket says 172.21.15.123:58816) 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.617+0000 7f3fd9e00640 1 -- 172.21.15.123:0/1919228468 learned_addr learned my addr 172.21.15.123:0/1919228468 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fda800640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 0x7f3fdc100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 -- 172.21.15.123:0/1919228468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 msgr2=0x7f3fdc100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 0x7f3fdc100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 -- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 msgr2=0x7f3fdc1000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc1000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 -- 172.21.15.123:0/1919228468 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc100ec0 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9400640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.618+0000 7f3fd9e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc000c00 tx=0x7f3fcc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.619+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc043020 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.619+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fcc0054e0 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.619+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc003c40 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.619+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc1a1af0 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.619+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc1a2090 con 0x7f3fdc102140 2024-03-24T00:35:10.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.620+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc0fd620 con 0x7f3fdc102140 2024-03-24T00:35:10.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.621+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3fcc046020 con 0x7f3fdc102140 2024-03-24T00:35:10.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.623+0000 7f3fd2a00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 0x7f3fc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.623+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3fcc0f0620 con 0x7f3fdc102140 2024-03-24T00:35:10.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.623+0000 7f3fd9400640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 0x7f3fc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.624+0000 7f3fd9400640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 0x7f3fc00a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc101820 tx=0x7f3fc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.627+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3fcc04d020 con 0x7f3fdc102140 2024-03-24T00:35:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:10 smithi177 ceph-mon[36034]: Deploying daemon mds.c on smithi204 2024-03-24T00:35:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:10 smithi177 ceph-mon[36034]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:10.863 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 podman[67492]: 2024-03-24 00:35:10.497245278 +0000 UTC m=+0.017802874 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:10.863 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 podman[67492]: 2024-03-24 00:35:10.610029085 +0000 UTC m=+0.130586653 container create f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T00:35:10.863 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 podman[67492]: 2024-03-24 00:35:10.837564215 +0000 UTC m=+0.358121794 container init f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:35:10.863 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 podman[67492]: 2024-03-24 00:35:10.840203452 +0000 UTC m=+0.360761040 container start f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T00:35:10.864 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 ceph-mds[67512]: -- 172.21.15.204:0/2991678360 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561b15ac1a40 con 0x561b15a95000 2024-03-24T00:35:10.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.865+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fdc02d070 con 0x7f3fdc102140 2024-03-24T00:35:10.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.866+0000 7f3fd2a00640 1 -- 172.21.15.123:0/1919228468 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5292 (secure 0 0 0) 0x7f3fcc0bdac0 con 0x7f3fdc102140 2024-03-24T00:35:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:10.866 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-24T00:35:10.866 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 7 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 msgr2=0x7f3fc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 0x7f3fc00a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc101820 tx=0x7f3fc8005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 msgr2=0x7f3fdc101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc000c00 tx=0x7f3fcc02c650 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 shutdown_connections 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc1073a0 0x7f3fdc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fc00a2880 0x7f3fc00a4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc104a70 0x7f3fdc1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 --2- 172.21.15.123:0/1919228468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fdc102140 0x7f3fdc101ab0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 >> 172.21.15.123:0/1919228468 conn(0x7f3fdc0fbb30 msgr2=0x7f3fdc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:10.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 shutdown_connections 2024-03-24T00:35:10.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:10.869+0000 7f3fe0e00640 1 -- 172.21.15.123:0/1919228468 wait complete. 2024-03-24T00:35:11.288 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c[67508]: starting mds.c at 2024-03-24T00:35:11.288 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 bash[67492]: f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 2024-03-24T00:35:11.288 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:35:10 smithi204 systemd[1]: Started Ceph mds.c for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1919228468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:11.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:11.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:11.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1919228468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:11.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:11.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:11.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:11.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1919228468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:11.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:11.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:11.679 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:11.679 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:12 smithi204 ceph-mon[36345]: Deploying daemon mds.h on smithi177 2024-03-24T00:35:12.588 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:12 smithi204 ceph-mon[36345]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:12.588 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:12 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.204:6836/1145111503,v1:172.21.15.204:6837/1145111503] up:boot 2024-03-24T00:35:12.588 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:12 smithi204 ceph-mon[36345]: fsmap 7 up:standby 2024-03-24T00:35:12.588 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:35:12.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:12 smithi123 ceph-mon[31487]: Deploying daemon mds.h on smithi177 2024-03-24T00:35:12.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:12 smithi123 ceph-mon[31487]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:12.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:12 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.204:6836/1145111503,v1:172.21.15.204:6837/1145111503] up:boot 2024-03-24T00:35:12.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:12 smithi123 ceph-mon[31487]: fsmap 7 up:standby 2024-03-24T00:35:12.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:35:12.707 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:12 smithi177 ceph-mon[36034]: Deploying daemon mds.h on smithi177 2024-03-24T00:35:12.707 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:12 smithi177 ceph-mon[36034]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:12.707 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:12 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.204:6836/1145111503,v1:172.21.15.204:6837/1145111503] up:boot 2024-03-24T00:35:12.708 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:12 smithi177 ceph-mon[36034]: fsmap 7 up:standby 2024-03-24T00:35:12.708 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:35:12.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 -- 172.21.15.123:0/3271701273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0930104ad0 msgr2=0x7f0930104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 --2- 172.21.15.123:0/3271701273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0930104ad0 0x7f0930104f50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f091c009900 tx=0x7f091c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 -- 172.21.15.123:0/3271701273 shutdown_connections 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 --2- 172.21.15.123:0/3271701273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 0x7f093010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 --2- 172.21.15.123:0/3271701273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0930104ad0 0x7f0930104f50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 --2- 172.21.15.123:0/3271701273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09301038e0 0x7f0930103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 -- 172.21.15.123:0/3271701273 >> 172.21.15.123:0/3271701273 conn(0x7f09300fee90 msgr2=0x7f09301012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:13.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 -- 172.21.15.123:0/3271701273 shutdown_connections 2024-03-24T00:35:13.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.101+0000 7f0934a00640 1 -- 172.21.15.123:0/3271701273 wait complete. 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.102+0000 7f0934a00640 1 Processor -- start 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.102+0000 7f0934a00640 1 -- start start 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.102+0000 7f0934a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.102+0000 7f0934a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 0x7f093019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.102+0000 7f0934a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 0x7f09301a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f0934a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093010f590 con 0x7f09301038e0 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f0934a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093010f410 con 0x7f0930105490 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f0934a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093010f710 con 0x7f0930104ad0 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60686/0 (socket says 172.21.15.123:60686) 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 -- 172.21.15.123:0/1321984172 learned_addr learned my addr 172.21.15.123:0/1321984172 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092e800640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 0x7f09301a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092d400640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 0x7f093019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 -- 172.21.15.123:0/1321984172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 msgr2=0x7f093019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 0x7f093019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 -- 172.21.15.123:0/1321984172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 msgr2=0x7f09301a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 0x7f09301a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092de00640 1 -- 172.21.15.123:0/1321984172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09301a17f0 con 0x7f09301038e0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.103+0000 7f092d400640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 0x7f093019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f092de00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f092000e9c0 tx=0x7f092000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000ccc0 con 0x7f09301038e0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09200035f0 con 0x7f09301038e0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09301a1ae0 con 0x7f09301038e0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000ce60 con 0x7f09301038e0 2024-03-24T00:35:13.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.104+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09301a2000 con 0x7f09301038e0 2024-03-24T00:35:13.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.105+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093010ac00 con 0x7f09301038e0 2024-03-24T00:35:13.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.108+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f09200037a0 con 0x7f09301038e0 2024-03-24T00:35:13.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.109+0000 7f0926a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.109+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09200ceaa0 con 0x7f09301038e0 2024-03-24T00:35:13.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.109+0000 7f092d400640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.110+0000 7f092d400640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 0x7f09140a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f091c02c240 tx=0x7f091c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:13.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.112+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f092001f340 con 0x7f09301038e0 2024-03-24T00:35:13.345 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 systemd[1]: Starting Ceph mds.h for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:13.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.360+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f093002d070 con 0x7f09301038e0 2024-03-24T00:35:13.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.361+0000 7f0926a00640 1 -- 172.21.15.123:0/1321984172 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6086 (secure 0 0 0) 0x7f092009bfe0 con 0x7f09301038e0 2024-03-24T00:35:13.361 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:13.362 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-24T00:35:13.362 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 msgr2=0x7f09140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 0x7f09140a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f091c02c240 tx=0x7f091c02df90 comp rx=0 tx=0).stop 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 msgr2=0x7f0930199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f092000e9c0 tx=0x7f092000edc0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 shutdown_connections 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0930105490 0x7f09301a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0930104ad0 0x7f093019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 --2- 172.21.15.123:0/1321984172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09301038e0 0x7f0930199bd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 >> 172.21.15.123:0/1321984172 conn(0x7f09300fee90 msgr2=0x7f0930107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:13.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.364+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 shutdown_connections 2024-03-24T00:35:13.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:13.365+0000 7f0934a00640 1 -- 172.21.15.123:0/1321984172 wait complete. 2024-03-24T00:35:13.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:13 smithi123 ceph-mon[31487]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:13.645 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:13 smithi177 ceph-mon[36034]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:13.646 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 podman[66674]: 2024-03-24 00:35:13.312544268 +0000 UTC m=+0.017275065 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:13.646 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 podman[66674]: 2024-03-24 00:35:13.435418274 +0000 UTC m=+0.140149044 container create 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T00:35:13.646 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 podman[66674]: 2024-03-24 00:35:13.64606737 +0000 UTC m=+0.350798161 container init 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T00:35:13.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:13 smithi204 ceph-mon[36345]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:13.962 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 podman[66674]: 2024-03-24 00:35:13.648629079 +0000 UTC m=+0.353359863 container start 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T00:35:13.962 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 ceph-mds[66694]: -- 172.21.15.177:0/3683054259 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c157fda40 con 0x560c157d1000 2024-03-24T00:35:13.963 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h[66690]: starting mds.h at 2024-03-24T00:35:13.963 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 bash[66674]: 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb 2024-03-24T00:35:13.963 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:35:13 smithi177 systemd[1]: Started Ceph mds.h for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1321984172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:14.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:14 smithi123 ceph-mon[31487]: Deploying daemon mds.i on smithi204 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1321984172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:14.658 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:14 smithi204 ceph-mon[36345]: Deploying daemon mds.i on smithi204 2024-03-24T00:35:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1321984172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:14 smithi177 ceph-mon[36034]: Deploying daemon mds.i on smithi204 2024-03-24T00:35:15.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 -- 172.21.15.123:0/3789153496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c107b00 msgr2=0x7f7c5c107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 --2- 172.21.15.123:0/3789153496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c107b00 0x7f7c5c107f20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c4402bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 -- 172.21.15.123:0/3789153496 shutdown_connections 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 --2- 172.21.15.123:0/3789153496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 --2- 172.21.15.123:0/3789153496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c108cf0 0x7f7c5c109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 --2- 172.21.15.123:0/3789153496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c107b00 0x7f7c5c107f20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 -- 172.21.15.123:0/3789153496 >> 172.21.15.123:0/3789153496 conn(0x7f7c5c076ab0 msgr2=0x7f7c5c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 -- 172.21.15.123:0/3789153496 shutdown_connections 2024-03-24T00:35:15.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.602+0000 7f7c62000640 1 -- 172.21.15.123:0/3789153496 wait complete. 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.603+0000 7f7c62000640 1 Processor -- start 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.603+0000 7f7c62000640 1 -- start start 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c108cf0 0x7f7c5c19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1137a0 con 0x7f7c5c108cf0 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c113620 con 0x7f7c5c107b00 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c62000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c113920 con 0x7f7c5c1096b0 2024-03-24T00:35:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c53e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c108cf0 0x7f7c5c19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56994/0 (socket says 172.21.15.123:56994) 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 -- 172.21.15.123:0/1937630026 learned_addr learned my addr 172.21.15.123:0/1937630026 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c61600640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 -- 172.21.15.123:0/1937630026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 msgr2=0x7f7c5c1a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c1a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 -- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c108cf0 msgr2=0x7f7c5c19e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c108cf0 0x7f7c5c19e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c60c00640 1 -- 172.21.15.123:0/1937630026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c1a5a30 con 0x7f7c5c107b00 2024-03-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.604+0000 7f7c61600640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.605+0000 7f7c60c00640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c4402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.605+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44043020 con 0x7f7c5c107b00 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.605+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c440054e0 con 0x7f7c5c107b00 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.605+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c1a5cc0 con 0x7f7c5c107b00 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.605+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44003c40 con 0x7f7c5c107b00 2024-03-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.606+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c1a61c0 con 0x7f7c5c107b00 2024-03-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.607+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c10ee20 con 0x7f7c5c107b00 2024-03-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.609+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7c44046020 con 0x7f7c5c107b00 2024-03-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.610+0000 7f7c5a000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 0x7f7c3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.610+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c440f0620 con 0x7f7c5c107b00 2024-03-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.611+0000 7f7c53e00640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 0x7f7c3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.611+0000 7f7c53e00640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 0x7f7c3c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c19f320 tx=0x7f7c4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.614+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c4404d020 con 0x7f7c5c107b00 2024-03-24T00:35:15.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:15 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6836/3793891487,v1:172.21.15.177:6837/3793891487] up:boot 2024-03-24T00:35:15.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:15 smithi177 ceph-mon[36034]: fsmap 8 up:standby 2024-03-24T00:35:15.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:15 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:35:15.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:15 smithi177 ceph-mon[36034]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:15.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:15 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6836/3793891487,v1:172.21.15.177:6837/3793891487] up:boot 2024-03-24T00:35:15.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:15 smithi204 ceph-mon[36345]: fsmap 8 up:standby 2024-03-24T00:35:15.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:15 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:35:15.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:15 smithi204 ceph-mon[36345]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:15.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.854+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c0622a0 con 0x7f7c5c107b00 2024-03-24T00:35:15.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.855+0000 7f7c5a000640 1 -- 172.21.15.123:0/1937630026 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6880 (secure 0 0 0) 0x7f7c440bdac0 con 0x7f7c5c107b00 2024-03-24T00:35:15.855 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:15.855 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-24T00:35:15.855 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 9 2024-03-24T00:35:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:15 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6836/3793891487,v1:172.21.15.177:6837/3793891487] up:boot 2024-03-24T00:35:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:15 smithi123 ceph-mon[31487]: fsmap 8 up:standby 2024-03-24T00:35:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:35:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:15 smithi123 ceph-mon[31487]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 msgr2=0x7f7c3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 0x7f7c3c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c19f320 tx=0x7f7c4c005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 msgr2=0x7f7c5c19de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c4402c650 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 shutdown_connections 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c5c1096b0 0x7f7c5c1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7c3c0a2800 0x7f7c3c0a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c5c108cf0 0x7f7c5c19e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 --2- 172.21.15.123:0/1937630026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c5c107b00 0x7f7c5c19de10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 >> 172.21.15.123:0/1937630026 conn(0x7f7c5c076ab0 msgr2=0x7f7c5c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 shutdown_connections 2024-03-24T00:35:15.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:15.858+0000 7f7c62000640 1 -- 172.21.15.123:0/1937630026 wait complete. 2024-03-24T00:35:16.071 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:15 smithi204 systemd[1]: Starting Ceph mds.i for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:16.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:16 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1937630026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:16.433 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 podman[67798]: 2024-03-24 00:35:16.067108769 +0000 UTC m=+0.017975234 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:16.433 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 podman[67798]: 2024-03-24 00:35:16.1971563 +0000 UTC m=+0.148022752 container create 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-24T00:35:16.433 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 podman[67798]: 2024-03-24 00:35:16.407920376 +0000 UTC m=+0.358786838 container init 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:35:16.434 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 podman[67798]: 2024-03-24 00:35:16.410760573 +0000 UTC m=+0.361627019 container start 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:35:16.434 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 ceph-mds[67817]: -- 172.21.15.204:0/244649704 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eb81a1fa40 con 0x55eb819f2800 2024-03-24T00:35:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:16 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1937630026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:16.821 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i[67813]: starting mds.i at 2024-03-24T00:35:16.821 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 bash[67798]: 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f 2024-03-24T00:35:16.821 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:35:16 smithi204 systemd[1]: Started Ceph mds.i for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:16 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1937630026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: Deploying daemon mds.g on smithi123 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.204:6838/3993860651,v1:172.21.15.204:6839/3993860651] up:boot 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: fsmap 9 up:standby 2024-03-24T00:35:17.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:17.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:17.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: Deploying daemon mds.g on smithi123 2024-03-24T00:35:17.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:17.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.204:6838/3993860651,v1:172.21.15.204:6839/3993860651] up:boot 2024-03-24T00:35:17.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: fsmap 9 up:standby 2024-03-24T00:35:17.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:35:17.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: Deploying daemon mds.g on smithi123 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.204:6838/3993860651,v1:172.21.15.204:6839/3993860651] up:boot 2024-03-24T00:35:17.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: fsmap 9 up:standby 2024-03-24T00:35:17.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:35:18.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 -- 172.21.15.123:0/3595704149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c107ce0 msgr2=0x7f0b7c108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 --2- 172.21.15.123:0/3595704149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c108100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009900 tx=0x7f0b6c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 -- 172.21.15.123:0/3595704149 shutdown_connections 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 --2- 172.21.15.123:0/3595704149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c109890 0x7f0b7c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 --2- 172.21.15.123:0/3595704149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 0x7f0b7c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 --2- 172.21.15.123:0/3595704149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c108100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.087+0000 7f0b81e00640 1 -- 172.21.15.123:0/3595704149 >> 172.21.15.123:0/3595704149 conn(0x7f0b7c076cb0 msgr2=0x7f0b7c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.088+0000 7f0b81e00640 1 -- 172.21.15.123:0/3595704149 shutdown_connections 2024-03-24T00:35:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.088+0000 7f0b81e00640 1 -- 172.21.15.123:0/3595704149 wait complete. 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.088+0000 7f0b81e00640 1 Processor -- start 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 -- start start 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 0x7f0b7c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c109890 0x7f0b7c1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c113980 con 0x7f0b7c109890 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c113800 con 0x7f0b7c107ce0 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b81e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c113b00 con 0x7f0b7c108ed0 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b7b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c109890 0x7f0b7c1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b7aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b7aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:58698/0 (socket says 172.21.15.123:58698) 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b7aa00640 1 -- 172.21.15.123:0/4152111555 learned_addr learned my addr 172.21.15.123:0/4152111555 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.089+0000 7f0b7a000640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 0x7f0b7c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 -- 172.21.15.123:0/4152111555 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 msgr2=0x7f0b7c19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 0x7f0b7c19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 -- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c109890 msgr2=0x7f0b7c1a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c109890 0x7f0b7c1a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 -- 172.21.15.123:0/4152111555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c1a5be0 con 0x7f0b7c107ce0 2024-03-24T00:35:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b7aa00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.090+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c043020 con 0x7f0b7c107ce0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.091+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b6c0054e0 con 0x7f0b7c107ce0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.091+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c1a5e70 con 0x7f0b7c107ce0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.091+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c003c40 con 0x7f0b7c107ce0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.091+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c1a6330 con 0x7f0b7c107ce0 2024-03-24T00:35:18.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.092+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c10efa0 con 0x7f0b7c107ce0 2024-03-24T00:35:18.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.094+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0b6c046020 con 0x7f0b7c107ce0 2024-03-24T00:35:18.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.095+0000 7f0b80a00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 0x7f0b600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.095+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0b6c0f0af0 con 0x7f0b7c107ce0 2024-03-24T00:35:18.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.095+0000 7f0b7a000640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 0x7f0b600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.096+0000 7f0b7a000640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 0x7f0b600a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b680095f0 tx=0x7f0b68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:18.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.097+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0b6c04d020 con 0x7f0b7c107ce0 2024-03-24T00:35:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.338+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7c02d070 con 0x7f0b7c107ce0 2024-03-24T00:35:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.339+0000 7f0b80a00640 1 -- 172.21.15.123:0/4152111555 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7676 (secure 0 0 0) 0x7f0b6c0bdf90 con 0x7f0b7c107ce0 2024-03-24T00:35:18.339 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:18.340 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:35:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 10 2024-03-24T00:35:18.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.342+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 msgr2=0x7f0b600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.342+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 0x7f0b600a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b680095f0 tx=0x7f0b68005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.342+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 msgr2=0x7f0b7c19dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.342+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c02c650 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 shutdown_connections 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0b600a2880 0x7f0b600a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b7c109890 0x7f0b7c1a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b7c108ed0 0x7f0b7c19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 --2- 172.21.15.123:0/4152111555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c107ce0 0x7f0b7c19dfc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 >> 172.21.15.123:0/4152111555 conn(0x7f0b7c076cb0 msgr2=0x7f0b7c10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 shutdown_connections 2024-03-24T00:35:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:18.343+0000 7f0b81e00640 1 -- 172.21.15.123:0/4152111555 wait complete. 2024-03-24T00:35:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:18 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4152111555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:18.989 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:18 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4152111555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:19.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:18 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4152111555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:19.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:19 smithi177 ceph-mon[36034]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:20.046 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:19 smithi123 ceph-mon[31487]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:20.046 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:19 smithi123 systemd[1]: Starting Ceph mds.g for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:20.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:19 smithi204 ceph-mon[36345]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:20.348 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 podman[72727]: 2024-03-24 00:35:19.94990057 +0000 UTC m=+0.034507955 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:20.348 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 podman[72727]: 2024-03-24 00:35:20.059608771 +0000 UTC m=+0.144216179 container create 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:35:20.348 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 podman[72727]: 2024-03-24 00:35:20.346966203 +0000 UTC m=+0.431573631 container init 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T00:35:20.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:20.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 -- 172.21.15.123:0/98861978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81016b0 msgr2=0x7f3bd8101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/98861978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81016b0 0x7f3bd8101ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009900 tx=0x7f3bcc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 -- 172.21.15.123:0/98861978 shutdown_connections 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/98861978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 0x7f3bd8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/98861978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81028a0 0x7f3bd8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/98861978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81016b0 0x7f3bd8101ad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 -- 172.21.15.123:0/98861978 >> 172.21.15.123:0/98861978 conn(0x7f3bd80fcc80 msgr2=0x7f3bd80ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 -- 172.21.15.123:0/98861978 shutdown_connections 2024-03-24T00:35:20.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.563+0000 7f3bdfc00640 1 -- 172.21.15.123:0/98861978 wait complete. 2024-03-24T00:35:20.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.564+0000 7f3bdfc00640 1 Processor -- start 2024-03-24T00:35:20.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.564+0000 7f3bdfc00640 1 -- start start 2024-03-24T00:35:20.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81028a0 0x7f3bd8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 0x7f3bd819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd810d380 con 0x7f3bd81016b0 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd810d200 con 0x7f3bd81028a0 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdfc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd810d500 con 0x7f3bd8103260 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54556/0 (socket says 172.21.15.123:54556) 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 -- 172.21.15.123:0/4115994086 learned_addr learned my addr 172.21.15.123:0/4115994086 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bd7e00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81028a0 0x7f3bd8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bddc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 0x7f3bd819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 -- 172.21.15.123:0/4115994086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 msgr2=0x7f3bd819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 0x7f3bd819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 -- 172.21.15.123:0/4115994086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81028a0 msgr2=0x7f3bd8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81028a0 0x7f3bd8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.565+0000 7f3bdd200640 1 -- 172.21.15.123:0/4115994086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd819f5c0 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bdd200640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc000c00 tx=0x7f3bcc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc043020 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd819f850 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bcc0054e0 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc003c40 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.566+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd819fda0 con 0x7f3bd81016b0 2024-03-24T00:35:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.567+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd8108970 con 0x7f3bd81016b0 2024-03-24T00:35:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.569+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3bcc046020 con 0x7f3bd81016b0 2024-03-24T00:35:20.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.571+0000 7f3bd5600640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 0x7f3bc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.571+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3bcc0f0600 con 0x7f3bd81016b0 2024-03-24T00:35:20.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.571+0000 7f3bd7e00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 0x7f3bc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.571+0000 7f3bd7e00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 0x7f3bc00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40095f0 tx=0x7f3bc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.575+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3bcc04d020 con 0x7f3bd81016b0 2024-03-24T00:35:20.606 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 podman[72727]: 2024-03-24 00:35:20.355765797 +0000 UTC m=+0.440373182 container start 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:35:20.606 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 ceph-mds[72745]: -- 172.21.15.123:0/1938878321 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621b1f51a40 con 0x5621b1f24c00 2024-03-24T00:35:20.606 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g[72741]: starting mds.g at 2024-03-24T00:35:20.606 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 bash[72727]: 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 2024-03-24T00:35:20.606 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:35:20 smithi123 systemd[1]: Started Ceph mds.g for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:20.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.823+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bd802d290 con 0x7f3bd81016b0 2024-03-24T00:35:20.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.825+0000 7f3bd5600640 1 -- 172.21.15.123:0/4115994086 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8471 (secure 0 0 0) 0x7f3bcc0bdaa0 con 0x7f3bd81016b0 2024-03-24T00:35:20.825 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:20.825 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-24T00:35:20.825 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 11 2024-03-24T00:35:20.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.827+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 msgr2=0x7f3bc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.827+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 0x7f3bc00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40095f0 tx=0x7f3bc4005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.827+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 msgr2=0x7f3bd81979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.827+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc000c00 tx=0x7f3bcc02c650 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 shutdown_connections 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bd8103260 0x7f3bd819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd81028a0 0x7f3bd8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bc00a2800 0x7f3bc00a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 --2- 172.21.15.123:0/4115994086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd81016b0 0x7f3bd81979a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 >> 172.21.15.123:0/4115994086 conn(0x7f3bd80fcc80 msgr2=0x7f3bd8105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 shutdown_connections 2024-03-24T00:35:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:20.828+0000 7f3bdfc00640 1 -- 172.21.15.123:0/4115994086 wait complete. 2024-03-24T00:35:21.798 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.798 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.798 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.798 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: Deploying daemon mds.a on smithi123 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.123:6838/1841887169,v1:172.21.15.123:6839/1841887169] up:boot 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: fsmap 10 up:standby 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4115994086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:21.799 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:21 smithi123 ceph-mon[31487]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: Deploying daemon mds.a on smithi123 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.123:6838/1841887169,v1:172.21.15.123:6839/1841887169] up:boot 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: fsmap 10 up:standby 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4115994086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:21.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:21 smithi204 ceph-mon[36345]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: Deploying daemon mds.a on smithi123 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.123:6838/1841887169,v1:172.21.15.123:6839/1841887169] up:boot 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: fsmap 10 up:standby 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4115994086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:21.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:21 smithi177 ceph-mon[36034]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:22.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 -- 172.21.15.123:0/2933244431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 msgr2=0x7f4878103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 --2- 172.21.15.123:0/2933244431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878103d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4868009900 tx=0x7f486802bc40 comp rx=0 tx=0).stop 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 -- 172.21.15.123:0/2933244431 shutdown_connections 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 --2- 172.21.15.123:0/2933244431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878105490 0x7f487810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 --2- 172.21.15.123:0/2933244431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878104ad0 0x7f4878104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 --2- 172.21.15.123:0/2933244431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878103d00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 -- 172.21.15.123:0/2933244431 >> 172.21.15.123:0/2933244431 conn(0x7f48780fee90 msgr2=0x7f48781012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 -- 172.21.15.123:0/2933244431 shutdown_connections 2024-03-24T00:35:23.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.270+0000 7f487d000640 1 -- 172.21.15.123:0/2933244431 wait complete. 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 Processor -- start 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 -- start start 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 0x7f487819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 0x7f48781a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487810f580 con 0x7f4878105490 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487810f400 con 0x7f48781038e0 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.271+0000 7f487d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487810f700 con 0x7f4878104ad0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4876800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 0x7f48781a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4876800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 0x7f48781a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54576/0 (socket says 172.21.15.123:54576) 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4876800640 1 -- 172.21.15.123:0/4213141285 learned_addr learned my addr 172.21.15.123:0/4213141285 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 0x7f487819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875e00640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 -- 172.21.15.123:0/4213141285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 msgr2=0x7f4878199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 -- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 msgr2=0x7f48781a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 0x7f48781a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 -- 172.21.15.123:0/4213141285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48781a1830 con 0x7f4878104ad0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.272+0000 7f4875400640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 0x7f487819a160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f486400e9f0 tx=0x7f486400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.273+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486400ccc0 con 0x7f4878104ad0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.273+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4864010910 con 0x7f4878104ad0 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.273+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48781a1b20 con 0x7f4878104ad0 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.273+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864018960 con 0x7f4878104ad0 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.273+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48781a2040 con 0x7f4878104ad0 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.274+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487810ac00 con 0x7f4878104ad0 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.276+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4864010ac0 con 0x7f4878104ad0 2024-03-24T00:35:23.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.277+0000 7f486ea00640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 0x7f485c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.278+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f48640ceaa0 con 0x7f4878104ad0 2024-03-24T00:35:23.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.278+0000 7f4875e00640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 0x7f485c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.278+0000 7f4875e00640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 0x7f485c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f486802c180 tx=0x7f486802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.281+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f486409b8f0 con 0x7f4878104ad0 2024-03-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.522+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48780622a0 con 0x7f4878104ad0 2024-03-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.524+0000 7f486ea00640 1 -- 172.21.15.123:0/4213141285 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8471 (secure 0 0 0) 0x7f486409f990 con 0x7f4878104ad0 2024-03-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 11 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 msgr2=0x7f485c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 0x7f485c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f486802c180 tx=0x7f486802cf90 comp rx=0 tx=0).stop 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 msgr2=0x7f487819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 0x7f487819a160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f486400e9f0 tx=0x7f486400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 shutdown_connections 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f485c0a2800 0x7f485c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4878105490 0x7f48781a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4878104ad0 0x7f487819a160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 --2- 172.21.15.123:0/4213141285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48781038e0 0x7f4878199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 >> 172.21.15.123:0/4213141285 conn(0x7f48780fee90 msgr2=0x7f4878107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 shutdown_connections 2024-03-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:23.526+0000 7f487d000640 1 -- 172.21.15.123:0/4213141285 wait complete. 2024-03-24T00:35:24.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:23 smithi123 podman[73111]: 2024-03-24 00:35:23.838573603 +0000 UTC m=+0.138064205 container create e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:35:24.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:24 smithi177 ceph-mon[36034]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:24.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:24 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4213141285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:24 smithi204 ceph-mon[36345]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:24 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4213141285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:24 smithi123 ceph-mon[31487]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:24 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4213141285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 podman[73111]: 2024-03-24 00:35:24.135239936 +0000 UTC m=+0.434730548 container init e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 podman[73111]: 2024-03-24 00:35:24.145248807 +0000 UTC m=+0.444739420 container start e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 ceph-mds[73130]: -- 172.21.15.123:0/1600992794 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558176249a40 con 0x55817621cc00 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: starting mds.a at 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 bash[73111]: e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a 2024-03-24T00:35:24.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:35:24 smithi123 systemd[1]: Started Ceph mds.a for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:25.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:25.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: Deploying daemon mds.b on smithi177 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.123:6840/829451039,v1:172.21.15.123:6841/829451039] up:boot 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: fsmap 11 up:standby 2024-03-24T00:35:25.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:35:25.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: Deploying daemon mds.b on smithi177 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.123:6840/829451039,v1:172.21.15.123:6841/829451039] up:boot 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: fsmap 11 up:standby 2024-03-24T00:35:25.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:35:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.775+0000 7f6026a00640 1 -- 172.21.15.123:0/656428453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020104a70 msgr2=0x7f6020106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.775+0000 7f6026a00640 1 --2- 172.21.15.123:0/656428453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020104a70 0x7f6020106e60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f600c009be0 tx=0x7f600c02bef0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 -- 172.21.15.123:0/656428453 shutdown_connections 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 --2- 172.21.15.123:0/656428453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60201073a0 0x7f60201097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 --2- 172.21.15.123:0/656428453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020104a70 0x7f6020106e60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 --2- 172.21.15.123:0/656428453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020102140 0x7f6020104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 -- 172.21.15.123:0/656428453 >> 172.21.15.123:0/656428453 conn(0x7f60200fbb30 msgr2=0x7f60200fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 -- 172.21.15.123:0/656428453 shutdown_connections 2024-03-24T00:35:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.776+0000 7f6026a00640 1 -- 172.21.15.123:0/656428453 wait complete. 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 Processor -- start 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 -- start start 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6020104a70 0x7f602019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 0x7f60201a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602010d380 con 0x7f60201073a0 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602010d200 con 0x7f6020102140 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.777+0000 7f6026a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602010d500 con 0x7f6020104a70 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f6024a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 0x7f60201a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:58770/0 (socket says 172.21.15.123:58770) 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f6024a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 0x7f60201a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54628/0 (socket says 172.21.15.123:54628) 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 -- 172.21.15.123:0/2610999919 learned_addr learned my addr 172.21.15.123:0/2610999919 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601f400640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6020104a70 0x7f602019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 -- 172.21.15.123:0/2610999919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6020104a70 msgr2=0x7f602019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6020104a70 0x7f602019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 -- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 msgr2=0x7f60201a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 0x7f60201a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 -- 172.21.15.123:0/2610999919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60201a1810 con 0x7f6020102140 2024-03-24T00:35:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601fe00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9c0 tx=0x7f601000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.778+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601000cde0 con 0x7f6020102140 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.779+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60100035f0 con 0x7f6020102140 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.779+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60201a1b00 con 0x7f6020102140 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.779+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010010690 con 0x7f6020102140 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.779+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60201a2070 con 0x7f6020102140 2024-03-24T00:35:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.780+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6020108ef0 con 0x7f6020102140 2024-03-24T00:35:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.782+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f60100037a0 con 0x7f6020102140 2024-03-24T00:35:25.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.783+0000 7f601cc00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 0x7f60040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.783+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60100ce8e0 con 0x7f6020102140 2024-03-24T00:35:25.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.783+0000 7f601f400640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 0x7f60040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.784+0000 7f601f400640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 0x7f60040a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f600c034fd0 tx=0x7f600c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:25.787+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f601009bd80 con 0x7f6020102140 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: Deploying daemon mds.b on smithi177 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.123:6840/829451039,v1:172.21.15.123:6841/829451039] up:boot 2024-03-24T00:35:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: fsmap 11 up:standby 2024-03-24T00:35:25.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:35:26.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.026+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f602002d070 con 0x7f6020102140 2024-03-24T00:35:26.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.028+0000 7f601cc00640 1 -- 172.21.15.123:0/2610999919 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9263 (secure 0 0 0) 0x7f602002d070 con 0x7f6020102140 2024-03-24T00:35:26.028 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:26.028 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-24T00:35:26.028 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 12 2024-03-24T00:35:26.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 msgr2=0x7f60040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:26.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 0x7f60040a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f600c034fd0 tx=0x7f600c000f40 comp rx=0 tx=0).stop 2024-03-24T00:35:26.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 msgr2=0x7f6020199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9c0 tx=0x7f601000ee80 comp rx=0 tx=0).stop 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 shutdown_connections 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f60040a2880 0x7f60040a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60201073a0 0x7f60201a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6020104a70 0x7f602019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 --2- 172.21.15.123:0/2610999919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6020102140 0x7f6020199bf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 >> 172.21.15.123:0/2610999919 conn(0x7f60200fbb30 msgr2=0x7f6020101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 shutdown_connections 2024-03-24T00:35:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:26.030+0000 7f6026a00640 1 -- 172.21.15.123:0/2610999919 wait complete. 2024-03-24T00:35:26.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:26 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2610999919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:26.713 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:26 smithi177 systemd[1]: Starting Ceph mds.b for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:35:26.722 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:26 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2610999919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:26 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2610999919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:27.024 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:26 smithi177 podman[66984]: 2024-03-24 00:35:26.674604225 +0000 UTC m=+0.017632984 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:27.024 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:26 smithi177 podman[66984]: 2024-03-24 00:35:26.779919211 +0000 UTC m=+0.122947989 container create 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T00:35:27.024 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:26 smithi177 podman[66984]: 2024-03-24 00:35:26.999080685 +0000 UTC m=+0.342109445 container init 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T00:35:27.025 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:27 smithi177 podman[66984]: 2024-03-24 00:35:27.001886093 +0000 UTC m=+0.344914844 container start 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:35:27.462 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:27 smithi177 ceph-mds[67003]: -- 172.21.15.177:0/1329197563 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ab5259a40 con 0x558ab522c800 2024-03-24T00:35:27.462 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: starting mds.b at 2024-03-24T00:35:27.462 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:27 smithi177 bash[66984]: 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 2024-03-24T00:35:27.462 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:27 smithi177 systemd[1]: Started Ceph mds.b for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:35:28.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.255+0000 7f964f400640 1 -- 172.21.15.123:0/1297561230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9648105490 msgr2=0x7f964810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.255+0000 7f964f400640 1 --2- 172.21.15.123:0/1297561230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9648105490 0x7f964810bd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9640009ca0 tx=0x7f964002c100 comp rx=0 tx=0).stop 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 -- 172.21.15.123:0/1297561230 shutdown_connections 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 --2- 172.21.15.123:0/1297561230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9648105490 0x7f964810bd20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 --2- 172.21.15.123:0/1297561230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648104ad0 0x7f9648104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 --2- 172.21.15.123:0/1297561230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96481038e0 0x7f9648103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 -- 172.21.15.123:0/1297561230 >> 172.21.15.123:0/1297561230 conn(0x7f96480fee90 msgr2=0x7f96481012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 -- 172.21.15.123:0/1297561230 shutdown_connections 2024-03-24T00:35:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.256+0000 7f964f400640 1 -- 172.21.15.123:0/1297561230 wait complete. 2024-03-24T00:35:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 Processor -- start 2024-03-24T00:35:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 -- start start 2024-03-24T00:35:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 0x7f9648199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 0x7f96481a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964810f560 con 0x7f9648104ad0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964810f3e0 con 0x7f9648105490 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964f400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964810f6e0 con 0x7f96481038e0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f9647e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f9647e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44326/0 (socket says 172.21.15.123:44326) 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f9647e00640 1 -- 172.21.15.123:0/1515866600 learned_addr learned my addr 172.21.15.123:0/1515866600 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964d400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 0x7f96481a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.257+0000 7f964ca00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 0x7f9648199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 -- 172.21.15.123:0/1515866600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 msgr2=0x7f9648199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 0x7f9648199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 -- 172.21.15.123:0/1515866600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 msgr2=0x7f96481a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 0x7f96481a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 -- 172.21.15.123:0/1515866600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96481a17c0 con 0x7f9648104ad0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f964d400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 0x7f96481a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f964ca00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 0x7f9648199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9647e00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f963400e9f0 tx=0x7f963400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.258+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963400cd80 con 0x7f9648104ad0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.259+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96481a1ab0 con 0x7f9648104ad0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.259+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9634005500 con 0x7f9648104ad0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.259+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634010690 con 0x7f9648104ad0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.259+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96481a1fb0 con 0x7f9648104ad0 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.260+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964810ac00 con 0x7f9648104ad0 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.263+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96340046c0 con 0x7f9648104ad0 2024-03-24T00:35:28.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.264+0000 7f9645600640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 0x7f96300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.265+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96340cea50 con 0x7f9648104ad0 2024-03-24T00:35:28.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.265+0000 7f964ca00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 0x7f96300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.265+0000 7f964ca00640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 0x7f96300a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9648104930 tx=0x7f963c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.269+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f963409bef0 con 0x7f9648104ad0 2024-03-24T00:35:28.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:28.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:boot 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: fsmap 12 up:standby 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:35:28.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:28 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.456 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:boot 2024-03-24T00:35:28.457 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: fsmap 12 up:standby 2024-03-24T00:35:28.457 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:35:28.457 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:28 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.523+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96480622a0 con 0x7f9648104ad0 2024-03-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.524+0000 7f9645600640 1 -- 172.21.15.123:0/1515866600 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10058 (secure 0 0 0) 0x7f963409b7f0 con 0x7f9648104ad0 2024-03-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 13 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 msgr2=0x7f96300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 0x7f96300a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9648104930 tx=0x7f963c005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 msgr2=0x7f964819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f963400e9f0 tx=0x7f963400edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 shutdown_connections 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9648105490 0x7f96481a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f96300a2800 0x7f96300a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9648104ad0 0x7f964819a0e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 --2- 172.21.15.123:0/1515866600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96481038e0 0x7f9648199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 >> 172.21.15.123:0/1515866600 conn(0x7f96480fee90 msgr2=0x7f9648107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 shutdown_connections 2024-03-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.527+0000 7f964f400640 1 -- 172.21.15.123:0/1515866600 wait complete. 2024-03-24T00:35:28.542 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:28.542 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:boot 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: fsmap 12 up:standby 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:35:28.543 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:28 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:28.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 -- 172.21.15.123:0/404969008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 msgr2=0x7f116c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 --2- 172.21.15.123:0/404969008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c109170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1154009900 tx=0x7f115402bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 -- 172.21.15.123:0/404969008 shutdown_connections 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 --2- 172.21.15.123:0/404969008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c1096b0 0x7f116c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 --2- 172.21.15.123:0/404969008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c109170 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 --2- 172.21.15.123:0/404969008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c107b00 0x7f116c107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 -- 172.21.15.123:0/404969008 >> 172.21.15.123:0/404969008 conn(0x7f116c076ab0 msgr2=0x7f116c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.738+0000 7f116be00640 1 -- 172.21.15.123:0/404969008 shutdown_connections 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.739+0000 7f116be00640 1 -- 172.21.15.123:0/404969008 wait complete. 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 Processor -- start 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 -- start start 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c19e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c1096b0 0x7f116c1a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1137b0 con 0x7f116c107b00 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c113630 con 0x7f116c108cf0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c113930 con 0x7f116c1096b0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44338/0 (socket says 172.21.15.123:44338) 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.740+0000 7f116aa00640 1 -- 172.21.15.123:0/2079612620 learned_addr learned my addr 172.21.15.123:0/2079612620 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116a000640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c19e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116b400640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c1096b0 0x7f116c1a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 -- 172.21.15.123:0/2079612620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c1096b0 msgr2=0x7f116c1a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c1096b0 0x7f116c1a5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 -- 172.21.15.123:0/2079612620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 msgr2=0x7f116c19e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c19e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 -- 172.21.15.123:0/2079612620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c1a59e0 con 0x7f116c107b00 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116a000640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c19e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116aa00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f115c00e9c0 tx=0x7f115c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00cd80 con 0x7f116c107b00 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c1a5cd0 con 0x7f116c107b00 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f115c0035f0 con 0x7f116c107b00 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.741+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c010690 con 0x7f116c107b00 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.742+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c1a6190 con 0x7f116c107b00 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.743+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116c10ee20 con 0x7f116c107b00 2024-03-24T00:35:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.744+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f115c0037a0 con 0x7f116c107b00 2024-03-24T00:35:28.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.746+0000 7f1163400640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 0x7f11500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.746+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f115c0ceac0 con 0x7f116c107b00 2024-03-24T00:35:28.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.746+0000 7f116a000640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 0x7f11500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.747+0000 7f116a000640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 0x7f11500a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f115402c240 tx=0x7f115402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.750+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f115c09bf60 con 0x7f116c107b00 2024-03-24T00:35:28.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.954+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f116c0623a0 con 0x7f116c107b00 2024-03-24T00:35:28.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.956+0000 7f1163400640 1 -- 172.21.15.123:0/2079612620 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18515 (secure 0 0 0) 0x7f115c09b860 con 0x7f116c107b00 2024-03-24T00:35:28.957 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:28.957 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:34:34.673592+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:28.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.958+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 msgr2=0x7f11500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.958+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 0x7f11500a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f115402c240 tx=0x7f115402df90 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.958+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 msgr2=0x7f116c19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.958+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f115c00e9c0 tx=0x7f115c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 shutdown_connections 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f116c1096b0 0x7f116c1a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c108cf0 0x7f116c19e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f11500a2800 0x7f11500a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 --2- 172.21.15.123:0/2079612620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f116c107b00 0x7f116c19ddc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 >> 172.21.15.123:0/2079612620 conn(0x7f116c076ab0 msgr2=0x7f116c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 shutdown_connections 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:28.959+0000 7f116be00640 1 -- 172.21.15.123:0/2079612620 wait complete. 2024-03-24T00:35:28.987 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:28.987 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:29.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 -- 172.21.15.123:0/3829019523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700fff10 msgr2=0x7fc970102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 --2- 172.21.15.123:0/3829019523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700fff10 0x7fc970102340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc960009ca0 tx=0x7fc96002c100 comp rx=0 tx=0).stop 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 -- 172.21.15.123:0/3829019523 shutdown_connections 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 --2- 172.21.15.123:0/3829019523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc970105230 0x7fc9701056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 --2- 172.21.15.123:0/3829019523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc970104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 --2- 172.21.15.123:0/3829019523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700fff10 0x7fc970102340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 -- 172.21.15.123:0/3829019523 >> 172.21.15.123:0/3829019523 conn(0x7fc9700fbb30 msgr2=0x7fc9700fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 -- 172.21.15.123:0/3829019523 shutdown_connections 2024-03-24T00:35:29.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.177+0000 7fc975a00640 1 -- 172.21.15.123:0/3829019523 wait complete. 2024-03-24T00:35:29.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.178+0000 7fc975a00640 1 Processor -- start 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 -- start start 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc97019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 0x7fc9701a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc970113910 con 0x7fc9700fff10 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc970113790 con 0x7fc970102880 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc975a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc970113a90 con 0x7fc970105230 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc96ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc966000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc97019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc96ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44344/0 (socket says 172.21.15.123:44344) 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc96f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 0x7fc9701a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc96ea00640 1 -- 172.21.15.123:0/1810802244 learned_addr learned my addr 172.21.15.123:0/1810802244 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.179+0000 7fc966000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc97019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46846/0 (socket says 172.21.15.123:46846) 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 -- 172.21.15.123:0/1810802244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 msgr2=0x7fc97019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc97019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 -- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 msgr2=0x7fc97019e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 -- 172.21.15.123:0/1810802244 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9701a5c00 con 0x7fc970105230 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96ea00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc96f400640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 0x7fc9701a5590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc96800e970 tx=0x7fc96800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96800ccc0 con 0x7fc970105230 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc968005020 con 0x7fc970105230 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9701a5ef0 con 0x7fc970105230 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968004a00 con 0x7fc970105230 2024-03-24T00:35:29.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.180+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9701a63b0 con 0x7fc970105230 2024-03-24T00:35:29.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.182+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9700fd620 con 0x7fc970105230 2024-03-24T00:35:29.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.183+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc968004ba0 con 0x7fc970105230 2024-03-24T00:35:29.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.184+0000 7fc967e00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 0x7fc9540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.185+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc9680ce580 con 0x7fc970105230 2024-03-24T00:35:29.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.185+0000 7fc96ea00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 0x7fc9540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.185+0000 7fc96ea00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 0x7fc9540a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc960006010 tx=0x7fc960004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:29.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.188+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc96809ba20 con 0x7fc970105230 2024-03-24T00:35:29.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.405+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc97002d070 con 0x7fc970105230 2024-03-24T00:35:29.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.406+0000 7fc967e00640 1 -- 172.21.15.123:0/1810802244 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9680b9e20 con 0x7fc970105230 2024-03-24T00:35:29.407 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 msgr2=0x7fc9540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 0x7fc9540a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc960006010 tx=0x7fc960004a40 comp rx=0 tx=0).stop 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 msgr2=0x7fc9701a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 0x7fc9701a5590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc96800e970 tx=0x7fc96800ee30 comp rx=0 tx=0).stop 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 shutdown_connections 2024-03-24T00:35:29.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970105230 0x7fc9701a5590 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc970102880 0x7fc97019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc9540a2880 0x7fc9540a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 --2- 172.21.15.123:0/1810802244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9700fff10 0x7fc97019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 >> 172.21.15.123:0/1810802244 conn(0x7fc9700fbb30 msgr2=0x7fc970103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 shutdown_connections 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.409+0000 7fc975a00640 1 -- 172.21.15.123:0/1810802244 wait complete. 2024-03-24T00:35:29.429 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:29 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1515866600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.429 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:29 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2079612620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.429 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.430 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:29 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:35:29.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:29 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1515866600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:29 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2079612620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:29 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:29 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1515866600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:29 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2079612620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:29 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 -- 172.21.15.123:0/4157569649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630105490 msgr2=0x7fe63010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 --2- 172.21.15.123:0/4157569649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630105490 0x7fe63010bd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe628009860 tx=0x7fe62802bb70 comp rx=0 tx=0).stop 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 -- 172.21.15.123:0/4157569649 shutdown_connections 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 --2- 172.21.15.123:0/4157569649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630105490 0x7fe63010bd20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 --2- 172.21.15.123:0/4157569649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe630104ad0 0x7fe630104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 --2- 172.21.15.123:0/4157569649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6301038e0 0x7fe630103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.627+0000 7fe635600640 1 -- 172.21.15.123:0/4157569649 >> 172.21.15.123:0/4157569649 conn(0x7fe6300fee90 msgr2=0x7fe6301012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.628+0000 7fe635600640 1 -- 172.21.15.123:0/4157569649 shutdown_connections 2024-03-24T00:35:29.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.628+0000 7fe635600640 1 -- 172.21.15.123:0/4157569649 wait complete. 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.628+0000 7fe635600640 1 Processor -- start 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 -- start start 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 0x7fe630199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe630105490 0x7fe6301a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63010f560 con 0x7fe630105490 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63010f3e0 con 0x7fe630104ad0 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe635600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63010f6e0 con 0x7fe6301038e0 2024-03-24T00:35:29.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe62e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe62e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46856/0 (socket says 172.21.15.123:46856) 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe62e000640 1 -- 172.21.15.123:0/1341925999 learned_addr learned my addr 172.21.15.123:0/1341925999 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe62ea00640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 0x7fe630199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.629+0000 7fe62f400640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe630105490 0x7fe6301a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 -- 172.21.15.123:0/1341925999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 msgr2=0x7fe630199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 0x7fe630199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 -- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe630105490 msgr2=0x7fe6301a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe630105490 0x7fe6301a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 -- 172.21.15.123:0/1341925999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6301a1790 con 0x7fe630104ad0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62ea00640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 0x7fe630199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe62e000640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e9f0 tx=0x7fe61800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61800cd80 con 0x7fe630104ad0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe618005500 con 0x7fe630104ad0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6301a1a80 con 0x7fe630104ad0 2024-03-24T00:35:29.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618010690 con 0x7fe630104ad0 2024-03-24T00:35:29.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.630+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6301a1f80 con 0x7fe630104ad0 2024-03-24T00:35:29.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.632+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63010ac00 con 0x7fe630104ad0 2024-03-24T00:35:29.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.634+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe6180046c0 con 0x7fe630104ad0 2024-03-24T00:35:29.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.635+0000 7fe627400640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 0x7fe6140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.635+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe6180cea50 con 0x7fe630104ad0 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.635+0000 7fe62ea00640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 0x7fe6140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.636+0000 7fe62ea00640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 0x7fe6140a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe6200098d0 tx=0x7fe620010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:29.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.639+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe61809bef0 con 0x7fe630104ad0 2024-03-24T00:35:29.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.848+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe63002d070 con 0x7fe630104ad0 2024-03-24T00:35:29.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.849+0000 7fe627400640 1 -- 172.21.15.123:0/1341925999 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe61809b7f0 con 0x7fe630104ad0 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:35:29.850 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-03-24T00:35:29.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.851+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 msgr2=0x7fe6140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.851+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 0x7fe6140a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe6200098d0 tx=0x7fe620010040 comp rx=0 tx=0).stop 2024-03-24T00:35:29.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 msgr2=0x7fe63019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:29.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e9f0 tx=0x7fe61800edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 shutdown_connections 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6140a2800 0x7fe6140a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe630105490 0x7fe6301a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630104ad0 0x7fe63019a0b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 --2- 172.21.15.123:0/1341925999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6301038e0 0x7fe630199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 >> 172.21.15.123:0/1341925999 conn(0x7fe6300fee90 msgr2=0x7fe630107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 shutdown_connections 2024-03-24T00:35:29.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:29.852+0000 7fe635600640 1 -- 172.21.15.123:0/1341925999 wait complete. 2024-03-24T00:35:29.871 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:35:29.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 -- 172.21.15.123:0/1352471811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881016b0 msgr2=0x7f7888101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 --2- 172.21.15.123:0/1352471811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881016b0 0x7f7888101ad0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f787c009900 tx=0x7f787c02bc50 comp rx=0 tx=0).stop 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 -- 172.21.15.123:0/1352471811 shutdown_connections 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 --2- 172.21.15.123:0/1352471811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 0x7f7888109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 --2- 172.21.15.123:0/1352471811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881028a0 0x7f7888102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 --2- 172.21.15.123:0/1352471811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881016b0 0x7f7888101ad0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 -- 172.21.15.123:0/1352471811 >> 172.21.15.123:0/1352471811 conn(0x7f78880fcc40 msgr2=0x7f78880ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 -- 172.21.15.123:0/1352471811 shutdown_connections 2024-03-24T00:35:30.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.087+0000 7f7890200640 1 -- 172.21.15.123:0/1352471811 wait complete. 2024-03-24T00:35:30.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 Processor -- start 2024-03-24T00:35:30.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 -- start start 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881016b0 0x7f78881979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 0x7f788819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810d370 con 0x7f78881028a0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810d1f0 con 0x7f7888103260 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.088+0000 7f7890200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810d4f0 con 0x7f78881016b0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44374/0 (socket says 172.21.15.123:44374) 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 -- 172.21.15.123:0/2145646352 learned_addr learned my addr 172.21.15.123:0/2145646352 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788d800640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881016b0 0x7f78881979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788e200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 0x7f788819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 -- 172.21.15.123:0/2145646352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881016b0 msgr2=0x7f78881979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881016b0 0x7f78881979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 -- 172.21.15.123:0/2145646352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 msgr2=0x7f788819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 0x7f788819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 -- 172.21.15.123:0/2145646352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788819f600 con 0x7f78881028a0 2024-03-24T00:35:30.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f788ce00640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f787400e9f0 tx=0x7f787400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.089+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787400cd80 con 0x7f78881028a0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.090+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788819f8f0 con 0x7f78881028a0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.090+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7874005500 con 0x7f78881028a0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.090+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874010690 con 0x7f78881028a0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.090+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788819fe40 con 0x7f78881028a0 2024-03-24T00:35:30.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.091+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888108a20 con 0x7f78881028a0 2024-03-24T00:35:30.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.095+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f78740046c0 con 0x7f78881028a0 2024-03-24T00:35:30.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.096+0000 7f7886000640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 0x7f78700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:30.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.096+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f78740ce9a0 con 0x7f78881028a0 2024-03-24T00:35:30.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.096+0000 7f788d800640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 0x7f78700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:30.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.097+0000 7f788d800640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 0x7f78700a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f787c02c190 tx=0x7f787c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:30.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.105+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f787409bee0 con 0x7f78881028a0 2024-03-24T00:35:30.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:30.311+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f788802d070 con 0x7f78881028a0 2024-03-24T00:35:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:30 smithi177 ceph-mon[36034]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:30 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:30 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1810802244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:35:30.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:30 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1341925999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:35:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:30 smithi204 ceph-mon[36345]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:30 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:30 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:30 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1810802244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:35:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:30 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1341925999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:35:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:30 smithi123 ceph-mon[31487]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:30.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:30 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:30.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:30 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1810802244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:35:30.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:30 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1341925999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:35:31.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.314+0000 7f7886000640 1 -- 172.21.15.123:0/2145646352 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7f787409b7e0 con 0x7f78881028a0 2024-03-24T00:35:31.315 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 msgr2=0x7f78700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 0x7f78700a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f787c02c190 tx=0x7f787c000bc0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 msgr2=0x7f7888197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f787400e9f0 tx=0x7f787400edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 shutdown_connections 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7888103260 0x7f788819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78700a2800 0x7f78700a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78881028a0 0x7f7888197f20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 --2- 172.21.15.123:0/2145646352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78881016b0 0x7f78881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.318+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 >> 172.21.15.123:0/2145646352 conn(0x7f78880fcc40 msgr2=0x7f7888105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:31.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.319+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 shutdown_connections 2024-03-24T00:35:31.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.319+0000 7f7890200640 1 -- 172.21.15.123:0/2145646352 wait complete. 2024-03-24T00:35:31.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.533+0000 7f8a77e00640 1 -- 172.21.15.123:0/3137381967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 msgr2=0x7f8a70109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.533+0000 7f8a77e00640 1 --2- 172.21.15.123:0/3137381967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a70109af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009ca0 tx=0x7f8a6802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 -- 172.21.15.123:0/3137381967 shutdown_connections 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 --2- 172.21.15.123:0/3137381967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a70109af0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 --2- 172.21.15.123:0/3137381967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701028a0 0x7f8a70102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 --2- 172.21.15.123:0/3137381967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701016b0 0x7f8a70101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 -- 172.21.15.123:0/3137381967 >> 172.21.15.123:0/3137381967 conn(0x7f8a700fcc40 msgr2=0x7f8a700ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 -- 172.21.15.123:0/3137381967 shutdown_connections 2024-03-24T00:35:31.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.534+0000 7f8a77e00640 1 -- 172.21.15.123:0/3137381967 wait complete. 2024-03-24T00:35:31.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.535+0000 7f8a77e00640 1 Processor -- start 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.535+0000 7f8a77e00640 1 -- start start 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.535+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701028a0 0x7f8a70197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a7019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a77e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7010d310 con 0x7f8a701028a0 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a77e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7010d190 con 0x7f8a701016b0 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a77e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7010d490 con 0x7f8a70103260 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46888/0 (socket says 172.21.15.123:46888) 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 -- 172.21.15.123:0/1481425007 learned_addr learned my addr 172.21.15.123:0/1481425007 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701028a0 0x7f8a70197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a7019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 -- 172.21.15.123:0/1481425007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 msgr2=0x7f8a7019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:31.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a7019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 -- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701028a0 msgr2=0x7f8a70197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701028a0 0x7f8a70197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 -- 172.21.15.123:0/1481425007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7019f550 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.536+0000 7f8a75400640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9c0 tx=0x7f8a6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.537+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400cd80 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.537+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a640035f0 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.537+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7019f840 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.537+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64010690 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.537+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7019fde0 con 0x7f8a701016b0 2024-03-24T00:35:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.538+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a701089d0 con 0x7f8a701016b0 2024-03-24T00:35:31.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.540+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8a640037a0 con 0x7f8a701016b0 2024-03-24T00:35:31.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.541+0000 7f8a6e000640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 0x7f8a580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:31.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.541+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f8a640cea80 con 0x7f8a701016b0 2024-03-24T00:35:31.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.541+0000 7f8a74a00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 0x7f8a580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:31.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.542+0000 7f8a74a00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 0x7f8a580a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a70198e40 tx=0x7f8a5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:31.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.545+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a6409be30 con 0x7f8a701016b0 2024-03-24T00:35:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:35:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:31.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:31.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:31 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:31.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:31.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:31 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:31.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:31.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:31 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:31.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:31.733+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8a7002d070 con 0x7f8a701016b0 2024-03-24T00:35:32.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.326+0000 7f8a6e000640 1 -- 172.21.15.123:0/1481425007 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8a6409b730 con 0x7f8a701016b0 2024-03-24T00:35:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-03-24T00:35:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.332+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 msgr2=0x7f8a580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.332+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 0x7f8a580a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a70198e40 tx=0x7f8a5c005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.332+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 msgr2=0x7f8a70197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.332+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9c0 tx=0x7f8a6400ee80 comp rx=0 tx=0).stop 2024-03-24T00:35:32.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 shutdown_connections 2024-03-24T00:35:32.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a70103260 0x7f8a7019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a580a2880 0x7f8a580a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a701028a0 0x7f8a70197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 --2- 172.21.15.123:0/1481425007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a701016b0 0x7f8a70197930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.333+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 >> 172.21.15.123:0/1481425007 conn(0x7f8a700fcc40 msgr2=0x7f8a70105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.334+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 shutdown_connections 2024-03-24T00:35:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.334+0000 7f8a77e00640 1 -- 172.21.15.123:0/1481425007 wait complete. 2024-03-24T00:35:32.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:35:32.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.541+0000 7f7e7b800640 1 -- 172.21.15.123:0/2954274340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 msgr2=0x7f7e74104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.541+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2954274340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e74104f50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009900 tx=0x7f7e6002bc50 comp rx=0 tx=0).stop 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.541+0000 7f7e7b800640 1 -- 172.21.15.123:0/2954274340 shutdown_connections 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.541+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2954274340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 0x7f7e7410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.541+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2954274340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e74104f50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.542+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2954274340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.542+0000 7f7e7b800640 1 -- 172.21.15.123:0/2954274340 >> 172.21.15.123:0/2954274340 conn(0x7f7e740fee50 msgr2=0x7f7e74101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.542+0000 7f7e7b800640 1 -- 172.21.15.123:0/2954274340 shutdown_connections 2024-03-24T00:35:32.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.542+0000 7f7e7b800640 1 -- 172.21.15.123:0/2954274340 wait complete. 2024-03-24T00:35:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 Processor -- start 2024-03-24T00:35:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 -- start start 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 0x7f7e741a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7410f5b0 con 0x7f7e74105490 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7410f430 con 0x7f7e741038e0 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e7b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7410f730 con 0x7f7e74104ad0 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e73e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e78e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e79800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 0x7f7e741a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e73e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56058/0 (socket says 172.21.15.123:56058) 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e78e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46906/0 (socket says 172.21.15.123:46906) 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.543+0000 7f7e73e00640 1 -- 172.21.15.123:0/2708842698 learned_addr learned my addr 172.21.15.123:0/2708842698 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 -- 172.21.15.123:0/2708842698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 msgr2=0x7f7e74199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 -- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 msgr2=0x7f7e741a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 0x7f7e741a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:32.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 -- 172.21.15.123:0/2708842698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e741a1850 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e73e00640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e6002c260 tx=0x7f7e60005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60041070 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e741a1ae0 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.544+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e600054e0 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.545+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600346a0 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.545+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741a2030 con 0x7f7e74104ad0 2024-03-24T00:35:32.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.546+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7410ac00 con 0x7f7e74104ad0 2024-03-24T00:35:32.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.548+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7e60044070 con 0x7f7e74104ad0 2024-03-24T00:35:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.549+0000 7f7e71600640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 0x7f7e5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.549+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f7e600f05e0 con 0x7f7e74104ad0 2024-03-24T00:35:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.549+0000 7f7e78e00640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 0x7f7e5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.549+0000 7f7e78e00640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 0x7f7e5c0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e7419b110 tx=0x7f7e6800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:32.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.553+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e60003e20 con 0x7f7e74104ad0 2024-03-24T00:35:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:35:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:32 smithi123 ceph-mon[31487]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:35:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1481425007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:32 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:32 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:35:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:32 smithi177 ceph-mon[36034]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:35:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:32 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1481425007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:32 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:32.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:32.792+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7e74062310 con 0x7f7e74104ad0 2024-03-24T00:35:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:32 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2145646352' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:35:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:32 smithi204 ceph-mon[36345]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:35:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:32 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1481425007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:32 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:35:33.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.353+0000 7f7e71600640 1 -- 172.21.15.123:0/2708842698 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7e600bd140 con 0x7f7e74104ad0 2024-03-24T00:35:33.354 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T00:35:33.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.358+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 msgr2=0x7f7e5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.358+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 0x7f7e5c0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e7419b110 tx=0x7f7e6800b390 comp rx=0 tx=0).stop 2024-03-24T00:35:33.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.358+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 msgr2=0x7f7e7419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.358+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e6002c260 tx=0x7f7e60005dd0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 shutdown_connections 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7e5c0a2800 0x7f7e5c0a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e74105490 0x7f7e741a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e74104ad0 0x7f7e7419a180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 --2- 172.21.15.123:0/2708842698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e741038e0 0x7f7e74199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 >> 172.21.15.123:0/2708842698 conn(0x7f7e740fee50 msgr2=0x7f7e74107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.359+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 shutdown_connections 2024-03-24T00:35:33.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.360+0000 7f7e7b800640 1 -- 172.21.15.123:0/2708842698 wait complete. 2024-03-24T00:35:33.382 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 -- 172.21.15.123:0/913289772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484104ad0 msgr2=0x7ff484104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 --2- 172.21.15.123:0/913289772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484104ad0 0x7ff484104f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff47002bc50 comp rx=0 tx=0).stop 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 -- 172.21.15.123:0/913289772 shutdown_connections 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 --2- 172.21.15.123:0/913289772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484105490 0x7ff48410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 --2- 172.21.15.123:0/913289772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484104ad0 0x7ff484104f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 --2- 172.21.15.123:0/913289772 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 -- 172.21.15.123:0/913289772 >> 172.21.15.123:0/913289772 conn(0x7ff4840fee90 msgr2=0x7ff4841012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 -- 172.21.15.123:0/913289772 shutdown_connections 2024-03-24T00:35:33.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.571+0000 7ff48c400640 1 -- 172.21.15.123:0/913289772 wait complete. 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 Processor -- start 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 -- start start 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 0x7ff484079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48410f590 con 0x7ff484105490 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48410f410 con 0x7ff484104ad0 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.572+0000 7ff48c400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48410f710 con 0x7ff4841038e0 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44422/0 (socket says 172.21.15.123:44422) 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 -- 172.21.15.123:0/3450188575 learned_addr learned my addr 172.21.15.123:0/3450188575 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff489000640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 0x7ff484079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff489a00640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 -- 172.21.15.123:0/3450188575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 msgr2=0x7ff484079450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484079450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 -- 172.21.15.123:0/3450188575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 msgr2=0x7ff484079990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 0x7ff484079990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 -- 172.21.15.123:0/3450188575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484076880 con 0x7ff484105490 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff489000640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 0x7ff484079990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff489a00640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.573+0000 7ff48a400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00efd0 tx=0x7ff47c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.574+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c003b70 con 0x7ff484105490 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.574+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff47c003d10 con 0x7ff484105490 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.574+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0109f0 con 0x7ff484105490 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.574+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4841a6020 con 0x7ff484105490 2024-03-24T00:35:33.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.574+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4841a63d0 con 0x7ff484105490 2024-03-24T00:35:33.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.575+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840688e0 con 0x7ff484105490 2024-03-24T00:35:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.577+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff47c010b90 con 0x7ff484105490 2024-03-24T00:35:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.579+0000 7ff482000640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 0x7ff46c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.579+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff47c0cf130 con 0x7ff484105490 2024-03-24T00:35:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.579+0000 7ff489a00640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 0x7ff46c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:33.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.579+0000 7ff489a00640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 0x7ff46c0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff484104930 tx=0x7ff478005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:33.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.585+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff47c09c280 con 0x7ff484105490 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2708842698' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:33 smithi123 ceph-mon[31487]: pgmap v312: 97 pgs: 16 creating+peering, 1 creating+activating, 11 active+clean, 69 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:35:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2708842698' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:33 smithi177 ceph-mon[36034]: pgmap v312: 97 pgs: 16 creating+peering, 1 creating+activating, 11 active+clean, 69 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:33.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.802+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff48402d200 con 0x7ff484105490 2024-03-24T00:35:33.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.806+0000 7ff482000640 1 -- 172.21.15.123:0/3450188575 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22291 (secure 0 0 0) 0x7ff47c09bb80 con 0x7ff484105490 2024-03-24T00:35:33.808 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:33.808 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:32.795547+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 msgr2=0x7ff46c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 0x7ff46c0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff484104930 tx=0x7ff478005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 msgr2=0x7ff484075fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00efd0 tx=0x7ff47c00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 shutdown_connections 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff46c0a2880 0x7ff46c0a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff484105490 0x7ff484075fc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff484104ad0 0x7ff484079990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 --2- 172.21.15.123:0/3450188575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4841038e0 0x7ff484079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.809+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 >> 172.21.15.123:0/3450188575 conn(0x7ff4840fee90 msgr2=0x7ff484107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.810+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 shutdown_connections 2024-03-24T00:35:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:33.810+0000 7ff48c400640 1 -- 172.21.15.123:0/3450188575 wait complete. 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2708842698' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:35:33.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:33.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:33 smithi204 ceph-mon[36345]: pgmap v312: 97 pgs: 16 creating+peering, 1 creating+activating, 11 active+clean, 69 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:35:33.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:34.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 -- 172.21.15.123:0/250564719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a4076060 msgr2=0x7f42a4076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 --2- 172.21.15.123:0/250564719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a4076060 0x7f42a4076480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4294009be0 tx=0x7f429402bef0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 -- 172.21.15.123:0/250564719 shutdown_connections 2024-03-24T00:35:34.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 --2- 172.21.15.123:0/250564719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 0x7f42a40784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 --2- 172.21.15.123:0/250564719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a40772a0 0x7f42a4077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 --2- 172.21.15.123:0/250564719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a4076060 0x7f42a4076480 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 -- 172.21.15.123:0/250564719 >> 172.21.15.123:0/250564719 conn(0x7f42a40fee50 msgr2=0x7f42a4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:34.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 -- 172.21.15.123:0/250564719 shutdown_connections 2024-03-24T00:35:34.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.019+0000 7f42ab800640 1 -- 172.21.15.123:0/250564719 wait complete. 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 Processor -- start 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 -- start start 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a4076060 0x7f42a419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 0x7f42a41a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4113970 con 0x7f42a40772a0 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41137f0 con 0x7f42a4076060 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.020+0000 7f42ab800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4113af0 con 0x7f42a4077c40 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44440/0 (socket says 172.21.15.123:44440) 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 -- 172.21.15.123:0/924847233 learned_addr learned my addr 172.21.15.123:0/924847233 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a9800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 0x7f42a41a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a8e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a4076060 0x7f42a419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 -- 172.21.15.123:0/924847233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 msgr2=0x7f42a41a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 0x7f42a41a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 -- 172.21.15.123:0/924847233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a4076060 msgr2=0x7f42a419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a4076060 0x7f42a419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 -- 172.21.15.123:0/924847233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a41a5bd0 con 0x7f42a40772a0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.021+0000 7f42a3e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f428c00e9f0 tx=0x7f428c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.022+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c00ccc0 con 0x7f42a40772a0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.022+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f428c010910 con 0x7f42a40772a0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.022+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41a5ec0 con 0x7f42a40772a0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.022+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c0188a0 con 0x7f42a40772a0 2024-03-24T00:35:34.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.022+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a41a63d0 con 0x7f42a40772a0 2024-03-24T00:35:34.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.023+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a41103b0 con 0x7f42a40772a0 2024-03-24T00:35:34.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.025+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f428c010ac0 con 0x7f42a40772a0 2024-03-24T00:35:34.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.026+0000 7f42a1600640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 0x7f42880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.026+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f428c0ced20 con 0x7f42a40772a0 2024-03-24T00:35:34.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.026+0000 7f42a8e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 0x7f42880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.027+0000 7f42a8e00640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 0x7f42880a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4294006010 tx=0x7f4294000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.031+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f428c09be70 con 0x7f42a40772a0 2024-03-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.235+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42a4062340 con 0x7f42a40772a0 2024-03-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.238+0000 7f42a1600640 1 -- 172.21.15.123:0/924847233 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22278 (secure 0 0 0) 0x7f428c09b770 con 0x7f42a40772a0 2024-03-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:34.240 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:33.818284+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 msgr2=0x7f42880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 0x7f42880a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4294006010 tx=0x7f4294000f40 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 msgr2=0x7f42a419e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f428c00e9f0 tx=0x7f428c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 shutdown_connections 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42a4077c40 0x7f42a41a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42880a2880 0x7f42880a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42a40772a0 0x7f42a419e4f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 --2- 172.21.15.123:0/924847233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42a4076060 0x7f42a419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 >> 172.21.15.123:0/924847233 conn(0x7f42a40fee50 msgr2=0x7f42a4100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:34.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 shutdown_connections 2024-03-24T00:35:34.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.241+0000 7f42ab800640 1 -- 172.21.15.123:0/924847233 wait complete. 2024-03-24T00:35:34.261 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 -- 172.21.15.123:0/2088013275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810101580 msgr2=0x7f48101019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 --2- 172.21.15.123:0/2088013275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810101580 0x7f48101019a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4800009900 tx=0x7f480002bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 -- 172.21.15.123:0/2088013275 shutdown_connections 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 --2- 172.21.15.123:0/2088013275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810103130 0x7f48101099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 --2- 172.21.15.123:0/2088013275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 --2- 172.21.15.123:0/2088013275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810101580 0x7f48101019a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 -- 172.21.15.123:0/2088013275 >> 172.21.15.123:0/2088013275 conn(0x7f48100fcb10 msgr2=0x7f48100fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 -- 172.21.15.123:0/2088013275 shutdown_connections 2024-03-24T00:35:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.476+0000 7f480fe00640 1 -- 172.21.15.123:0/2088013275 wait complete. 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.477+0000 7f480fe00640 1 Processor -- start 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 -- start start 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810101580 0x7f4810075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810103130 0x7f4810073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481010d240 con 0x7f4810101580 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481010d0c0 con 0x7f4810103130 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481010d3c0 con 0x7f4810102770 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810101580 0x7f4810075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56112/0 (socket says 172.21.15.123:56112) 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480e000640 1 -- 172.21.15.123:0/2297381625 learned_addr learned my addr 172.21.15.123:0/2297381625 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.478+0000 7f480f400640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810103130 0x7f4810073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 -- 172.21.15.123:0/2297381625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810103130 msgr2=0x7f4810073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810103130 0x7f4810073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 -- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810101580 msgr2=0x7f4810075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810101580 0x7f4810075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 -- 172.21.15.123:0/2297381625 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810074860 con 0x7f4810102770 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480e000640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e9f0 tx=0x7f47fc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00cd80 con 0x7f4810102770 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48101a19a0 con 0x7f4810102770 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47fc005500 con 0x7f4810102770 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc010690 con 0x7f4810102770 2024-03-24T00:35:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.479+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48101a1ef0 con 0x7f4810102770 2024-03-24T00:35:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.480+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4810108840 con 0x7f4810102770 2024-03-24T00:35:34.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.483+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f47fc0046c0 con 0x7f4810102770 2024-03-24T00:35:34.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.485+0000 7f4807400640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 0x7f47f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:34.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.485+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f47fc0cee40 con 0x7f4810102770 2024-03-24T00:35:34.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.485+0000 7f480ea00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 0x7f47f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:34.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.486+0000 7f480ea00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 0x7f47f40a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f480002c170 tx=0x7f480002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:34.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.486+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47fc09bf90 con 0x7f4810102770 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3450188575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:35:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:34 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/924847233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.715 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3450188575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:34.716 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:35:34.716 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:34 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/924847233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:34.725+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f481002d070 con 0x7f4810102770 2024-03-24T00:35:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:35:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:35:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3450188575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:35:34.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:34 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/924847233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:35.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.375+0000 7f4807400640 1 -- 172.21.15.123:0/2297381625 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f47fc084e20 con 0x7f4810102770 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 msgr2=0x7f47f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 0x7f47f40a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f480002c170 tx=0x7f480002df90 comp rx=0 tx=0).stop 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 msgr2=0x7f4810073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e9f0 tx=0x7f47fc00edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 shutdown_connections 2024-03-24T00:35:35.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4810103130 0x7f4810073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4810102770 0x7f4810073a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f47f40a2800 0x7f47f40a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 --2- 172.21.15.123:0/2297381625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4810101580 0x7f4810075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 >> 172.21.15.123:0/2297381625 conn(0x7f48100fcb10 msgr2=0x7f4810105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 shutdown_connections 2024-03-24T00:35:35.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.379+0000 7f480fe00640 1 -- 172.21.15.123:0/2297381625 wait complete. 2024-03-24T00:35:35.400 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-24T00:35:35.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T00:35:35.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 -- 172.21.15.123:0/494829726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4105490 msgr2=0x7f50e410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:35.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 --2- 172.21.15.123:0/494829726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4105490 0x7f50e410bd20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009900 tx=0x7f50dc02bc40 comp rx=0 tx=0).stop 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 -- 172.21.15.123:0/494829726 shutdown_connections 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 --2- 172.21.15.123:0/494829726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4105490 0x7f50e410bd20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 --2- 172.21.15.123:0/494829726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e4104ad0 0x7f50e4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 --2- 172.21.15.123:0/494829726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e41038e0 0x7f50e4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 -- 172.21.15.123:0/494829726 >> 172.21.15.123:0/494829726 conn(0x7f50e40fee50 msgr2=0x7f50e4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 -- 172.21.15.123:0/494829726 shutdown_connections 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.622+0000 7f50e9800640 1 -- 172.21.15.123:0/494829726 wait complete. 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 Processor -- start 2024-03-24T00:35:35.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 -- start start 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e41038e0 0x7f50e4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 0x7f50e41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410f5b0 con 0x7f50e4105490 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410f430 con 0x7f50e4104ad0 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e9800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410f730 con 0x7f50e41038e0 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50e3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 0x7f50e41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:35.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.623+0000 7f50da000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50e3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 0x7f50e41a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44486/0 (socket says 172.21.15.123:44486) 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50e3400640 1 -- 172.21.15.123:0/74516322 learned_addr learned my addr 172.21.15.123:0/74516322 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46966/0 (socket says 172.21.15.123:46966) 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50e2a00640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e41038e0 0x7f50e4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 -- 172.21.15.123:0/74516322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e41038e0 msgr2=0x7f50e4199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e41038e0 0x7f50e4199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 -- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 msgr2=0x7f50e41a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:35.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 0x7f50e41a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 -- 172.21.15.123:0/74516322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e41a17e0 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50da000640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00e9f0 tx=0x7f50cc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc00ccc0 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50cc005020 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0049e0 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.624+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e41a1ad0 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.625+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e41a2020 con 0x7f50e4104ad0 2024-03-24T00:35:35.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.626+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e410ac00 con 0x7f50e4104ad0 2024-03-24T00:35:35.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.627+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f50cc005560 con 0x7f50e4104ad0 2024-03-24T00:35:35.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.628+0000 7f50dbe00640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 0x7f50c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:35.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.629+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50cc0ce8b0 con 0x7f50e4104ad0 2024-03-24T00:35:35.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.629+0000 7f50e2a00640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 0x7f50c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:35.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.629+0000 7f50e2a00640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 0x7f50c80a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f50e4104930 tx=0x7f50d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:35.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.633+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f50cc0051d0 con 0x7f50e4104ad0 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:active 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2297381625' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:35 smithi177 ceph-mon[36034]: pgmap v314: 97 pgs: 16 creating+peering, 1 creating+activating, 55 active+clean, 25 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:active 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2297381625' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:35 smithi204 ceph-mon[36345]: pgmap v314: 97 pgs: 16 creating+peering, 1 creating+activating, 55 active+clean, 25 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] up:active 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2297381625' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:35:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:35 smithi123 ceph-mon[31487]: pgmap v314: 97 pgs: 16 creating+peering, 1 creating+activating, 55 active+clean, 25 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:35:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:35.873+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f50e40622a0 con 0x7f50e4104ad0 2024-03-24T00:35:36.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.036+0000 7f50dbe00640 1 -- 172.21.15.123:0/74516322 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7f50cc09baa0 con 0x7f50e4104ad0 2024-03-24T00:35:36.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.040+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 msgr2=0x7f50c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.040+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 0x7f50c80a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f50e4104930 tx=0x7f50d4009500 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.040+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 msgr2=0x7f50e419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.040+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00e9f0 tx=0x7f50cc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 shutdown_connections 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f50c80a2880 0x7f50c80a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4105490 0x7f50e41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50e4104ad0 0x7f50e419a100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 --2- 172.21.15.123:0/74516322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50e41038e0 0x7f50e4199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 >> 172.21.15.123:0/74516322 conn(0x7f50e40fee50 msgr2=0x7f50e4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 shutdown_connections 2024-03-24T00:35:36.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.041+0000 7f50e9800640 1 -- 172.21.15.123:0/74516322 wait complete. 2024-03-24T00:35:36.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.256+0000 7f563b400640 1 -- 172.21.15.123:0/4274800669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634107cc0 msgr2=0x7f56341080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.256+0000 7f563b400640 1 --2- 172.21.15.123:0/4274800669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634107cc0 0x7f56341080e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5624009900 tx=0x7f562402bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.256+0000 7f563b400640 1 -- 172.21.15.123:0/4274800669 shutdown_connections 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.256+0000 7f563b400640 1 --2- 172.21.15.123:0/4274800669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634109870 0x7f5634110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.257+0000 7f563b400640 1 --2- 172.21.15.123:0/4274800669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 0x7f5634109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.257+0000 7f563b400640 1 --2- 172.21.15.123:0/4274800669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634107cc0 0x7f56341080e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.257+0000 7f563b400640 1 -- 172.21.15.123:0/4274800669 >> 172.21.15.123:0/4274800669 conn(0x7f5634076c60 msgr2=0x7f56340790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.257+0000 7f563b400640 1 -- 172.21.15.123:0/4274800669 shutdown_connections 2024-03-24T00:35:36.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.257+0000 7f563b400640 1 -- 172.21.15.123:0/4274800669 wait complete. 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 Processor -- start 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 -- start start 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634107cc0 0x7f563419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 0x7f563419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634113960 con 0x7f5634108eb0 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56341137e0 con 0x7f5634109870 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f563b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634113ae0 con 0x7f5634107cc0 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f5639400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f5639400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46552/0 (socket says 172.21.15.123:46552) 2024-03-24T00:35:36.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f5639400640 1 -- 172.21.15.123:0/1138276572 learned_addr learned my addr 172.21.15.123:0/1138276572 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f562be00640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 0x7f563419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.258+0000 7f5638a00640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634107cc0 0x7f563419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 -- 172.21.15.123:0/1138276572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634107cc0 msgr2=0x7f563419e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634107cc0 0x7f563419e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 -- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 msgr2=0x7f563419e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 0x7f563419e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 -- 172.21.15.123:0/1138276572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56341a5c30 con 0x7f5634109870 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5639400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f562c009fd0 tx=0x7f562c00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c017070 con 0x7f5634109870 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f562c003e50 con 0x7f5634109870 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0106c0 con 0x7f5634109870 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56341a5f20 con 0x7f5634109870 2024-03-24T00:35:36.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.259+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56341a63e0 con 0x7f5634109870 2024-03-24T00:35:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.260+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563410f0b0 con 0x7f5634109870 2024-03-24T00:35:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.262+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f562c003970 con 0x7f5634109870 2024-03-24T00:35:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.263+0000 7f5632000640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 0x7f56180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.264+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f562c0ce650 con 0x7f5634109870 2024-03-24T00:35:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.264+0000 7f5638a00640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 0x7f56180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:36.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.264+0000 7f5638a00640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 0x7f56180a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f562402c170 tx=0x7f562402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.269+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f562c09b150 con 0x7f5634109870 2024-03-24T00:35:36.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:36.510+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f5634062340 con 0x7f5634109870 2024-03-24T00:35:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:35:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/74516322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:35:36.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:36 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/74516322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:35:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:36 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:35:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:35:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/74516322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:35:36.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:35:36.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:36 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.389+0000 7f5632000640 1 -- 172.21.15.123:0/1138276572 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f562c09f1f0 con 0x7f5634109870 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 msgr2=0x7f56180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 0x7f56180a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f562402c170 tx=0x7f562402df90 comp rx=0 tx=0).stop 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 msgr2=0x7f56341a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f562c009fd0 tx=0x7f562c00b260 comp rx=0 tx=0).stop 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 shutdown_connections 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634109870 0x7f56341a55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f56180a2880 0x7f56180a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634108eb0 0x7f563419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 --2- 172.21.15.123:0/1138276572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5634107cc0 0x7f563419e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 >> 172.21.15.123:0/1138276572 conn(0x7f5634076c60 msgr2=0x7f563410c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:37.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 shutdown_connections 2024-03-24T00:35:37.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.393+0000 7f563b400640 1 -- 172.21.15.123:0/1138276572 wait complete. 2024-03-24T00:35:37.418 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-24T00:35:37.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.640+0000 7fd3e1c00640 1 -- 172.21.15.123:0/3453179711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 msgr2=0x7fd3dc1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.640+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/3453179711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1018c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009900 tx=0x7fd3cc02bc50 comp rx=0 tx=0).stop 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 -- 172.21.15.123:0/3453179711 shutdown_connections 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/3453179711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/3453179711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 0x7fd3dc102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/3453179711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1018c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 -- 172.21.15.123:0/3453179711 >> 172.21.15.123:0/3453179711 conn(0x7fd3dc0fca50 msgr2=0x7fd3dc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 -- 172.21.15.123:0/3453179711 shutdown_connections 2024-03-24T00:35:37.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.641+0000 7fd3e1c00640 1 -- 172.21.15.123:0/3453179711 wait complete. 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 Processor -- start 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 -- start start 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 0x7fd3dc199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc10d150 con 0x7fd3dc1014a0 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc10cfd0 con 0x7fd3dc103050 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc10d2d0 con 0x7fd3dc102690 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3dbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46566/0 (socket says 172.21.15.123:46566) 2024-03-24T00:35:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.642+0000 7fd3e1200640 1 -- 172.21.15.123:0/2325003559 learned_addr learned my addr 172.21.15.123:0/2325003559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3db400640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 0x7fd3dc199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 -- 172.21.15.123:0/2325003559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 msgr2=0x7fd3dc199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 0x7fd3dc199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 -- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 msgr2=0x7fd3dc1999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 -- 172.21.15.123:0/2325003559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc1a15d0 con 0x7fd3dc103050 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3dbe00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1200640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400eb10 tx=0x7fd3d400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4004b20 con 0x7fd3dc103050 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3d4004cc0 con 0x7fd3dc103050 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc1a18c0 con 0x7fd3dc103050 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.643+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4010760 con 0x7fd3dc103050 2024-03-24T00:35:37.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.644+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc1a1e10 con 0x7fd3dc103050 2024-03-24T00:35:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.645+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc1a20a0 con 0x7fd3dc103050 2024-03-24T00:35:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.646+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd3d4005020 con 0x7fd3dc103050 2024-03-24T00:35:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.647+0000 7fd3d8c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 0x7fd3c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.648+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd3d40cee90 con 0x7fd3dc103050 2024-03-24T00:35:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.648+0000 7fd3dbe00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 0x7fd3c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:37.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.648+0000 7fd3dbe00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 0x7fd3c00a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc02c190 tx=0x7fd3cc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:37.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.651+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd3d409bfe0 con 0x7fd3dc103050 2024-03-24T00:35:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:37 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1138276572' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:37 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:37 smithi177 ceph-mon[36034]: pgmap v315: 97 pgs: 16 creating+peering, 1 creating+activating, 72 active+clean, 8 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 178 B/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-24T00:35:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:37 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1138276572' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:37 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:37 smithi204 ceph-mon[36345]: pgmap v315: 97 pgs: 16 creating+peering, 1 creating+activating, 72 active+clean, 8 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 178 B/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-24T00:35:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1138276572' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:35:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-mon[31487]: pgmap v315: 97 pgs: 16 creating+peering, 1 creating+activating, 72 active+clean, 8 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 178 B/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-24T00:35:37.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.877+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3dc1a2270 con 0x7fd3c00a2880 2024-03-24T00:35:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.900+0000 7fd3d8c00640 1 -- 172.21.15.123:0/2325003559 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd3dc1a2270 con 0x7fd3c00a2880 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 msgr2=0x7fd3c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 0x7fd3c00a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc02c190 tx=0x7fd3cc000bc0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 msgr2=0x7fd3dc1a0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400eb10 tx=0x7fd3d400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 shutdown_connections 2024-03-24T00:35:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3dc103050 0x7fd3dc1a0f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3dc102690 0x7fd3dc199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd3c00a2880 0x7fd3c00a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 --2- 172.21.15.123:0/2325003559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc1014a0 0x7fd3dc1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 >> 172.21.15.123:0/2325003559 conn(0x7fd3dc0fca50 msgr2=0x7fd3dc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 shutdown_connections 2024-03-24T00:35:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:37.902+0000 7fd3e1c00640 1 -- 172.21.15.123:0/2325003559 wait complete. 2024-03-24T00:35:37.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 -- 172.21.15.123:0/3738377779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34108db0 msgr2=0x7f6c34109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 --2- 172.21.15.123:0/3738377779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34108db0 0x7f6c34109230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009ca0 tx=0x7f6c1c02c100 comp rx=0 tx=0).stop 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 -- 172.21.15.123:0/3738377779 shutdown_connections 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 --2- 172.21.15.123:0/3738377779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34109770 0x7f6c34110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 --2- 172.21.15.123:0/3738377779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34108db0 0x7f6c34109230 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 --2- 172.21.15.123:0/3738377779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c34107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 -- 172.21.15.123:0/3738377779 >> 172.21.15.123:0/3738377779 conn(0x7f6c34076b70 msgr2=0x7f6c34078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 -- 172.21.15.123:0/3738377779 shutdown_connections 2024-03-24T00:35:38.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.124+0000 7f6c33200640 1 -- 172.21.15.123:0/3738377779 wait complete. 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 Processor -- start 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 -- start start 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c3419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 0x7f6c3419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 0x7f6c341a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34113840 con 0x7f6c34108db0 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c341136c0 con 0x7f6c34109770 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.125+0000 7f6c33200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c341139c0 con 0x7f6c34107bc0 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c31400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 0x7f6c3419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c31e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c3419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c31e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c3419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:45588/0 (socket says 172.21.15.123:45588) 2024-03-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c31e00640 1 -- 172.21.15.123:0/562918933 learned_addr learned my addr 172.21.15.123:0/562918933 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 0x7f6c341a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 -- 172.21.15.123:0/562918933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 msgr2=0x7f6c3419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c3419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 -- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 msgr2=0x7f6c3419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 0x7f6c3419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 -- 172.21.15.123:0/562918933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c341a5b00 con 0x7f6c34109770 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c31400640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 0x7f6c3419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c32800640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 0x7f6c341a5490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00d6b0 tx=0x7f6c2c00db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.126+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c017070 con 0x7f6c34109770 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.127+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c341a5df0 con 0x7f6c34109770 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.127+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c2c009d00 con 0x7f6c34109770 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.127+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c003d60 con 0x7f6c34109770 2024-03-24T00:35:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.127+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c341a6260 con 0x7f6c34109770 2024-03-24T00:35:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.128+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3410ef30 con 0x7f6c34109770 2024-03-24T00:35:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.130+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6c2c009820 con 0x7f6c34109770 2024-03-24T00:35:38.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.131+0000 7f6c2aa00640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 0x7f6c140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.131+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6c2c0ce190 con 0x7f6c34109770 2024-03-24T00:35:38.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.131+0000 7f6c31e00640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 0x7f6c140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.132+0000 7f6c31e00640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 0x7f6c140a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c240058c0 tx=0x7f6c24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:38.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.140+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c2c09ad10 con 0x7f6c34109770 2024-03-24T00:35:38.357 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:35:37.886+0000 7f0b7ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:35:38.357 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:35:37.886+0000 7f0b7ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:35:38.357 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:35:37.886+0000 7f0b7ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:35:38.357 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:35:37.886+0000 7f0b7ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:35:38.357 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:35:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:35:37.886+0000 7f0b7ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:35:38.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.370+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c34061b70 con 0x7f6c140a2800 2024-03-24T00:35:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.403+0000 7f6c2aa00640 1 -- 172.21.15.123:0/562918933 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6c34061b70 con 0x7f6c140a2800 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.405+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 msgr2=0x7f6c140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.405+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 0x7f6c140a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c240058c0 tx=0x7f6c24005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.405+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 msgr2=0x7f6c341a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.405+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 0x7f6c341a5490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00d6b0 tx=0x7f6c2c00db70 comp rx=0 tx=0).stop 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 shutdown_connections 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34109770 0x7f6c341a5490 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6c140a2800 0x7f6c140a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c34108db0 0x7f6c3419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 --2- 172.21.15.123:0/562918933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c34107bc0 0x7f6c3419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 >> 172.21.15.123:0/562918933 conn(0x7f6c34076b70 msgr2=0x7f6c3410c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 shutdown_connections 2024-03-24T00:35:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.406+0000 7f6c33200640 1 -- 172.21.15.123:0/562918933 wait complete. 2024-03-24T00:35:38.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T00:35:38.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 -- 172.21.15.123:0/498299934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8006a4b0 msgr2=0x7fcf8010b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 --2- 172.21.15.123:0/498299934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010b1b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009900 tx=0x7fcf7002bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 -- 172.21.15.123:0/498299934 shutdown_connections 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 --2- 172.21.15.123:0/498299934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8010e1e0 0x7fcf80110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 --2- 172.21.15.123:0/498299934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 0x7fcf8010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 --2- 172.21.15.123:0/498299934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010b1b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 -- 172.21.15.123:0/498299934 >> 172.21.15.123:0/498299934 conn(0x7fcf8006e8a0 msgr2=0x7fcf80070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 -- 172.21.15.123:0/498299934 shutdown_connections 2024-03-24T00:35:38.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.658+0000 7fcf86e00640 1 -- 172.21.15.123:0/498299934 wait complete. 2024-03-24T00:35:38.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 Processor -- start 2024-03-24T00:35:38.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 -- start start 2024-03-24T00:35:38.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 0x7fcf80109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80114aa0 con 0x7fcf8010b780 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80114920 con 0x7fcf8010e1e0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf86e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80114c20 con 0x7fcf8006a4b0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf84e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf84e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46624/0 (socket says 172.21.15.123:46624) 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf77400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 0x7fcf80109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.660+0000 7fcf84e00640 1 -- 172.21.15.123:0/3583340926 learned_addr learned my addr 172.21.15.123:0/3583340926 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf7fe00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 -- 172.21.15.123:0/3583340926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8006a4b0 msgr2=0x7fcf8010ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 -- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 msgr2=0x7fcf80109360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 0x7fcf80109360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 -- 172.21.15.123:0/3583340926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8010a130 con 0x7fcf8010e1e0 2024-03-24T00:35:38.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf84e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009fd0 tx=0x7fcf7800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78017070 con 0x7fcf8010e1e0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf78003e50 con 0x7fcf8010e1e0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf780106c0 con 0x7fcf8010e1e0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.661+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf801aa3c0 con 0x7fcf8010e1e0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.662+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf801aa910 con 0x7fcf8010e1e0 2024-03-24T00:35:38.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.663+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf801aabd0 con 0x7fcf8010e1e0 2024-03-24T00:35:38.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.664+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcf78003970 con 0x7fcf8010e1e0 2024-03-24T00:35:38.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.666+0000 7fcf7d600640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 0x7fcf640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:38.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.666+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcf780ce5d0 con 0x7fcf8010e1e0 2024-03-24T00:35:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.666+0000 7fcf7fe00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 0x7fcf640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.667+0000 7fcf7fe00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 0x7fcf640a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:38.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.670+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf7809b0d0 con 0x7fcf8010e1e0 2024-03-24T00:35:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:38 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:35:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:38 smithi177 ceph-mon[36034]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:38 smithi177 ceph-mon[36034]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:38.712 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:35:38 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: 2024-03-24T00:35:38.426+0000 7f3e1ae38700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:35:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:38 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:35:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:38 smithi204 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:38 smithi204 ceph-mon[36345]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:38 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:35:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:38 smithi123 ceph-mon[31487]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-24T00:35:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:38 smithi123 ceph-mon[31487]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.898+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf8006fc90 con 0x7fcf640a2880 2024-03-24T00:35:38.910 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/qa/sv_0/fbeb6ad5-4cdd-4c76-a10c-d69116e225da 2024-03-24T00:35:38.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.909+0000 7fcf7d600640 1 -- 172.21.15.123:0/3583340926 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fcf8006fc90 con 0x7fcf640a2880 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 msgr2=0x7fcf640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 0x7fcf640a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).stop 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 msgr2=0x7fcf801098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009fd0 tx=0x7fcf7800b260 comp rx=0 tx=0).stop 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 shutdown_connections 2024-03-24T00:35:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf8010e1e0 0x7fcf801098a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcf640a2880 0x7fcf640a4d40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf8010b780 0x7fcf80109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 --2- 172.21.15.123:0/3583340926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8006a4b0 0x7fcf8010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 >> 172.21.15.123:0/3583340926 conn(0x7fcf8006e8a0 msgr2=0x7fcf80111aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 shutdown_connections 2024-03-24T00:35:38.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:38.912+0000 7fcf86e00640 1 -- 172.21.15.123:0/3583340926 wait complete. 2024-03-24T00:35:38.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.149+0000 7faac5400640 1 -- 172.21.15.123:0/262753001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 msgr2=0x7faac0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.149+0000 7faac5400640 1 --2- 172.21.15.123:0/262753001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac0104f50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009900 tx=0x7faaa802bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 -- 172.21.15.123:0/262753001 shutdown_connections 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 --2- 172.21.15.123:0/262753001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 --2- 172.21.15.123:0/262753001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac0104f50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 --2- 172.21.15.123:0/262753001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 -- 172.21.15.123:0/262753001 >> 172.21.15.123:0/262753001 conn(0x7faac00fee50 msgr2=0x7faac0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 -- 172.21.15.123:0/262753001 shutdown_connections 2024-03-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.150+0000 7faac5400640 1 -- 172.21.15.123:0/262753001 wait complete. 2024-03-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 Processor -- start 2024-03-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 -- start start 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac010f5b0 con 0x7faac01038e0 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac010f430 con 0x7faac0104ad0 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faac5400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac010f730 con 0x7faac0105490 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faabe800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.151+0000 7faab7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabe800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33472/0 (socket says 172.21.15.123:33472) 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faab7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46638/0 (socket says 172.21.15.123:46638) 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabe800640 1 -- 172.21.15.123:0/2529454393 learned_addr learned my addr 172.21.15.123:0/2529454393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 -- 172.21.15.123:0/2529454393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 msgr2=0x7faac019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 -- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 msgr2=0x7faac0199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 -- 172.21.15.123:0/2529454393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac01a1830 con 0x7faac0105490 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faab7e00640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faabf200640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac01a1190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faab800e940 tx=0x7faab800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8004d20 con 0x7faac0105490 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faab8005020 con 0x7faac0105490 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.152+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac01a1b20 con 0x7faac0105490 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.153+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab80116d0 con 0x7faac0105490 2024-03-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.153+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac01a2070 con 0x7faac0105490 2024-03-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.154+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac010ac00 con 0x7faac0105490 2024-03-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.156+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faab80054d0 con 0x7faac0105490 2024-03-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.157+0000 7faab7400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 0x7faaa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.157+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faab80ced50 con 0x7faac0105490 2024-03-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.157+0000 7faabe800640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 0x7faaa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.158+0000 7faabe800640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 0x7faaa40a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faac019b0f0 tx=0x7faab0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.159+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faab8016030 con 0x7faac0105490 2024-03-24T00:35:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.388+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7faac0061ae0 con 0x7faaa40a2800 2024-03-24T00:35:39.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.405+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7faab8097e30 con 0x7faac0105490 2024-03-24T00:35:39.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.425+0000 7faab7400640 1 -- 172.21.15.123:0/2529454393 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faac0061ae0 con 0x7faaa40a2800 2024-03-24T00:35:39.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 msgr2=0x7faaa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 0x7faaa40a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faac019b0f0 tx=0x7faab0005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:39.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 msgr2=0x7faac01a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac01a1190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faab800e940 tx=0x7faab800ee00 comp rx=0 tx=0).stop 2024-03-24T00:35:39.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 shutdown_connections 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac0105490 0x7faac01a1190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac0104ad0 0x7faac019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faaa40a2800 0x7faaa40a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 --2- 172.21.15.123:0/2529454393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac01038e0 0x7faac0199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.428+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 >> 172.21.15.123:0/2529454393 conn(0x7faac00fee50 msgr2=0x7faac0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.429+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 shutdown_connections 2024-03-24T00:35:39.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.429+0000 7faac5400640 1 -- 172.21.15.123:0/2529454393 wait complete. 2024-03-24T00:35:39.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.653+0000 7f3a78000640 1 -- 172.21.15.123:0/3237404998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a70104ad0 msgr2=0x7f3a70104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.653+0000 7f3a78000640 1 --2- 172.21.15.123:0/3237404998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a70104ad0 0x7f3a70104f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009900 tx=0x7f3a5c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 -- 172.21.15.123:0/3237404998 shutdown_connections 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 --2- 172.21.15.123:0/3237404998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a7010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 --2- 172.21.15.123:0/3237404998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a70104ad0 0x7f3a70104f50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 --2- 172.21.15.123:0/3237404998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a701038e0 0x7f3a70103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 -- 172.21.15.123:0/3237404998 >> 172.21.15.123:0/3237404998 conn(0x7f3a700fee50 msgr2=0x7f3a70101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 -- 172.21.15.123:0/3237404998 shutdown_connections 2024-03-24T00:35:39.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.654+0000 7f3a78000640 1 -- 172.21.15.123:0/3237404998 wait complete. 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 Processor -- start 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 -- start start 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 0x7f3a7019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a701a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7010f5b0 con 0x7f3a70104ad0 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7010f430 con 0x7f3a701038e0 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.655+0000 7f3a78000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7010f730 con 0x7f3a70105490 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46650/0 (socket says 172.21.15.123:46650) 2024-03-24T00:35:39.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 -- 172.21.15.123:0/645026021 learned_addr learned my addr 172.21.15.123:0/645026021 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a74c00640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 0x7f3a7019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a76000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a701a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 -- 172.21.15.123:0/645026021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 msgr2=0x7f3a701a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a701a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 -- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 msgr2=0x7f3a7019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 0x7f3a7019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 -- 172.21.15.123:0/645026021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a701a1830 con 0x7f3a701038e0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a76000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a701a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a74c00640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 0x7f3a7019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a75600640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d950 tx=0x7f3a6400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64017020 con 0x7f3a701038e0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a64009c30 con 0x7f3a701038e0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400f720 con 0x7f3a701038e0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.656+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a701a1b20 con 0x7f3a701038e0 2024-03-24T00:35:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.657+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a701a2070 con 0x7f3a701038e0 2024-03-24T00:35:39.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.658+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7010ac00 con 0x7f3a701038e0 2024-03-24T00:35:39.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.659+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3a6401a020 con 0x7f3a701038e0 2024-03-24T00:35:39.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.661+0000 7f3a6e000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 0x7f3a580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:39.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.661+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3a640cebb0 con 0x7f3a701038e0 2024-03-24T00:35:39.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.661+0000 7f3a74c00640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 0x7f3a580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.662+0000 7f3a74c00640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 0x7f3a580a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c02c250 tx=0x7f3a5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:39.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.665+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a64021020 con 0x7f3a701038e0 2024-03-24T00:35:39.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:39 smithi177 ceph-mon[36034]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:39 smithi177 ceph-mon[36034]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:39 smithi177 ceph-mon[36034]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 303 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-24T00:35:39.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:39 smithi204 ceph-mon[36345]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:39 smithi204 ceph-mon[36345]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:39 smithi204 ceph-mon[36345]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 303 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-24T00:35:39.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:39 smithi123 ceph-mon[31487]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:39 smithi123 ceph-mon[31487]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:39.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:39 smithi123 ceph-mon[31487]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 303 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-24T00:35:39.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.894+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a70061ae0 con 0x7f3a580a2890 2024-03-24T00:35:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.905+0000 7f3a6e000640 1 -- 172.21.15.123:0/645026021 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f3a70061ae0 con 0x7f3a580a2890 2024-03-24T00:35:39.906 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/qa/sv_1/4b45e73b-8124-4842-a181-0a29172dc089 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 msgr2=0x7f3a580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 0x7f3a580a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c02c250 tx=0x7f3a5c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 msgr2=0x7f3a70199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d950 tx=0x7f3a6400de10 comp rx=0 tx=0).stop 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 shutdown_connections 2024-03-24T00:35:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a70105490 0x7f3a701a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a580a2890 0x7f3a580a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a70104ad0 0x7f3a7019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 --2- 172.21.15.123:0/645026021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a701038e0 0x7f3a70199c20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 >> 172.21.15.123:0/645026021 conn(0x7f3a700fee50 msgr2=0x7f3a70107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 shutdown_connections 2024-03-24T00:35:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:39.908+0000 7f3a78000640 1 -- 172.21.15.123:0/645026021 wait complete. 2024-03-24T00:35:39.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 -- 172.21.15.123:0/979594867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 msgr2=0x7fd9d8109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 --2- 172.21.15.123:0/979594867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d8109170 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009900 tx=0x7fd9c402bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 -- 172.21.15.123:0/979594867 shutdown_connections 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 --2- 172.21.15.123:0/979594867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d81096b0 0x7fd9d810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 --2- 172.21.15.123:0/979594867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d8109170 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 --2- 172.21.15.123:0/979594867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d8107b00 0x7fd9d8107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 -- 172.21.15.123:0/979594867 >> 172.21.15.123:0/979594867 conn(0x7fd9d8076ab0 msgr2=0x7fd9d8078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 -- 172.21.15.123:0/979594867 shutdown_connections 2024-03-24T00:35:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.140+0000 7fd9de800640 1 -- 172.21.15.123:0/979594867 wait complete. 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 Processor -- start 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 -- start start 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d8107b00 0x7fd9d819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d81096b0 0x7fd9d81a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d81137b0 con 0x7fd9d81096b0 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8113630 con 0x7fd9d8108cf0 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.141+0000 7fd9de800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8113930 con 0x7fd9d8107b00 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46670/0 (socket says 172.21.15.123:46670) 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 -- 172.21.15.123:0/691724059 learned_addr learned my addr 172.21.15.123:0/691724059 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dde00640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d81096b0 0x7fd9d81a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dd400640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d8107b00 0x7fd9d819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 -- 172.21.15.123:0/691724059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d8107b00 msgr2=0x7fd9d819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d8107b00 0x7fd9d819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 -- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d81096b0 msgr2=0x7fd9d81a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d81096b0 0x7fd9d81a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 -- 172.21.15.123:0/691724059 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d81a5a40 con 0x7fd9d8108cf0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9dca00640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.142+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4043020 con 0x7fd9d8108cf0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.143+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c40054e0 con 0x7fd9d8108cf0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.143+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4003bd0 con 0x7fd9d8108cf0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.143+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d81a5cd0 con 0x7fd9d8108cf0 2024-03-24T00:35:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.143+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d81a61d0 con 0x7fd9d8108cf0 2024-03-24T00:35:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.144+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d810ee70 con 0x7fd9d8108cf0 2024-03-24T00:35:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.145+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd9c4046020 con 0x7fd9d8108cf0 2024-03-24T00:35:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.147+0000 7fd9d6000640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 0x7fd9c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.147+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd9c40f0a80 con 0x7fd9d8108cf0 2024-03-24T00:35:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.147+0000 7fd9dd400640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 0x7fd9c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.148+0000 7fd9dd400640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 0x7fd9c00a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0095f0 tx=0x7fd9cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.151+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9c40f2e20 con 0x7fd9d8108cf0 2024-03-24T00:35:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.390+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d802d070 con 0x7fd9d8108cf0 2024-03-24T00:35:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.391+0000 7fd9d6000640 1 -- 172.21.15.123:0/691724059 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fd9c404d020 con 0x7fd9d8108cf0 2024-03-24T00:35:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:40.393 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:35:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:35:40.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.394+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 msgr2=0x7fd9c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.394+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 0x7fd9c00a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0095f0 tx=0x7fd9cc005f90 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.394+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 msgr2=0x7fd9d819e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.394+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c402c650 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 shutdown_connections 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd9c00a2910 0x7fd9c00a4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d81096b0 0x7fd9d81a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d8108cf0 0x7fd9d819e360 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 --2- 172.21.15.123:0/691724059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9d8107b00 0x7fd9d819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 >> 172.21.15.123:0/691724059 conn(0x7fd9d8076ab0 msgr2=0x7fd9d810c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 shutdown_connections 2024-03-24T00:35:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.395+0000 7fd9de800640 1 -- 172.21.15.123:0/691724059 wait complete. 2024-03-24T00:35:40.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 -- 172.21.15.123:0/1940648110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 msgr2=0x7f3704107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 --2- 172.21.15.123:0/1940648110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f3704107f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009860 tx=0x7f36f402bb70 comp rx=0 tx=0).stop 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 -- 172.21.15.123:0/1940648110 shutdown_connections 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 --2- 172.21.15.123:0/1940648110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37041096b0 0x7f370410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 --2- 172.21.15.123:0/1940648110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704108cf0 0x7f3704109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 --2- 172.21.15.123:0/1940648110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f3704107f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.618+0000 7f3709400640 1 -- 172.21.15.123:0/1940648110 >> 172.21.15.123:0/1940648110 conn(0x7f3704076ab0 msgr2=0x7f3704078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.619+0000 7f3709400640 1 -- 172.21.15.123:0/1940648110 shutdown_connections 2024-03-24T00:35:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.619+0000 7f3709400640 1 -- 172.21.15.123:0/1940648110 wait complete. 2024-03-24T00:35:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.619+0000 7f3709400640 1 Processor -- start 2024-03-24T00:35:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 -- start start 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f370419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 0x7f370419e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3704113750 con 0x7f3704108cf0 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041135d0 con 0x7f37041096b0 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3709400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041138d0 con 0x7f3704107b00 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3703e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f370419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f36fb400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 0x7f370419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3708a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3703e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f370419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:45662/0 (socket says 172.21.15.123:45662) 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3703e00640 1 -- 172.21.15.123:0/3485976671 learned_addr learned my addr 172.21.15.123:0/3485976671 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.620+0000 7f3708a00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46694/0 (socket says 172.21.15.123:46694) 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 -- 172.21.15.123:0/3485976671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 msgr2=0x7f370419ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f370419ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 -- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 msgr2=0x7f370419e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 0x7f370419e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 -- 172.21.15.123:0/3485976671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37041a5a10 con 0x7f37041096b0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f36fb400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 0x7f370419e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3708a00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009fd0 tx=0x7f36fc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc00ce20 con 0x7f37041096b0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36fc005020 con 0x7f37041096b0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37041a5d00 con 0x7f37041096b0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc010720 con 0x7f37041096b0 2024-03-24T00:35:40.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.621+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37041a61f0 con 0x7f37041096b0 2024-03-24T00:35:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.622+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370410ee20 con 0x7f37041096b0 2024-03-24T00:35:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.624+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f36fc005540 con 0x7f37041096b0 2024-03-24T00:35:40.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.625+0000 7f3701600640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 0x7f36e80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:40.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.625+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f36fc0cf140 con 0x7f37041096b0 2024-03-24T00:35:40.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.625+0000 7f3703e00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 0x7f36e80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:40.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.626+0000 7f3703e00640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 0x7f36e80a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f36f40095f0 tx=0x7f36f4034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:40.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.629+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f36fc09c170 con 0x7f37041096b0 2024-03-24T00:35:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:40 smithi177 ceph-mon[36034]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:40 smithi177 ceph-mon[36034]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T00:35:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:40 smithi177 ceph-mon[36034]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:40 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/691724059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:40 smithi204 ceph-mon[36345]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:40 smithi204 ceph-mon[36345]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T00:35:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:40 smithi204 ceph-mon[36345]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:40 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/691724059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.830+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37040622a0 con 0x7f37041096b0 2024-03-24T00:35:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.832+0000 7f3701600640 1 -- 172.21.15.123:0/3485976671 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22278 (secure 0 0 0) 0x7f36fc09ba70 con 0x7f37041096b0 2024-03-24T00:35:40.833 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:40.834 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:33.818284+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 msgr2=0x7f36e80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 0x7f36e80a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f36f40095f0 tx=0x7f36f4034ad0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 msgr2=0x7f37041a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009fd0 tx=0x7f36fc00ef10 comp rx=0 tx=0).stop 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 shutdown_connections 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041096b0 0x7f37041a53a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36e80a2910 0x7f36e80a4dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3704108cf0 0x7f370419e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 --2- 172.21.15.123:0/3485976671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3704107b00 0x7f370419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 >> 172.21.15.123:0/3485976671 conn(0x7f3704076ab0 msgr2=0x7f370410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 shutdown_connections 2024-03-24T00:35:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:40.835+0000 7f3709400640 1 -- 172.21.15.123:0/3485976671 wait complete. 2024-03-24T00:35:40.852 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:40 smithi123 ceph-mon[31487]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.852 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:40 smithi123 ceph-mon[31487]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T00:35:40.852 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:40 smithi123 ceph-mon[31487]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:35:40.852 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:40 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/691724059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:35:40.855 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:35:40.855 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:35:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 -- 172.21.15.123:0/3805421616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac41038e0 msgr2=0x7fbac4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 --2- 172.21.15.123:0/3805421616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac41038e0 0x7fbac4103d00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009900 tx=0x7fbab802bc30 comp rx=0 tx=0).stop 2024-03-24T00:35:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 -- 172.21.15.123:0/3805421616 shutdown_connections 2024-03-24T00:35:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 --2- 172.21.15.123:0/3805421616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac4105490 0x7fbac410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 --2- 172.21.15.123:0/3805421616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4104ad0 0x7fbac4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 --2- 172.21.15.123:0/3805421616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac41038e0 0x7fbac4103d00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 -- 172.21.15.123:0/3805421616 >> 172.21.15.123:0/3805421616 conn(0x7fbac40fee90 msgr2=0x7fbac41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 -- 172.21.15.123:0/3805421616 shutdown_connections 2024-03-24T00:35:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.062+0000 7fbacc800640 1 -- 172.21.15.123:0/3805421616 wait complete. 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.063+0000 7fbacc800640 1 Processor -- start 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 -- start start 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac41038e0 0x7fbac4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4105490 0x7fbac41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410f5b0 con 0x7fbac4105490 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410f430 con 0x7fbac4104ad0 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbacc800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410f730 con 0x7fbac41038e0 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbac9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbaca800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4105490 0x7fbac41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbac9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46708/0 (socket says 172.21.15.123:46708) 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbac9400640 1 -- 172.21.15.123:0/479324208 learned_addr learned my addr 172.21.15.123:0/479324208 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.064+0000 7fbac9e00640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac41038e0 0x7fbac4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 -- 172.21.15.123:0/479324208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac41038e0 msgr2=0x7fbac4199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac41038e0 0x7fbac4199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 -- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4105490 msgr2=0x7fbac41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4105490 0x7fbac41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 -- 172.21.15.123:0/479324208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac41a1810 con 0x7fbac4104ad0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbac9400640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbab400e9f0 tx=0x7fbab400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab400cd80 con 0x7fbac4104ad0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbab4010910 con 0x7fbac4104ad0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac41a1b00 con 0x7fbac4104ad0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4018990 con 0x7fbac4104ad0 2024-03-24T00:35:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.065+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac41a20a0 con 0x7fbac4104ad0 2024-03-24T00:35:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.067+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac410aba0 con 0x7fbac4104ad0 2024-03-24T00:35:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.069+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbab4010ac0 con 0x7fbac4104ad0 2024-03-24T00:35:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.070+0000 7fbabea00640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 0x7fbaac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.070+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbab40cedd0 con 0x7fbac4104ad0 2024-03-24T00:35:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.070+0000 7fbac9e00640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 0x7fbaac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.071+0000 7fbac9e00640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 0x7fbaac0a4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbab802c170 tx=0x7fbab802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.076+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbab409b770 con 0x7fbac4104ad0 2024-03-24T00:35:41.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.274+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbac4061ae0 con 0x7fbaac0a2910 2024-03-24T00:35:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.290+0000 7fbabea00640 1 -- 172.21.15.123:0/479324208 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258688 (secure 0 0 0) 0x7fbac4061ae0 con 0x7fbaac0a2910 2024-03-24T00:35:41.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 msgr2=0x7fbaac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 0x7fbaac0a4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbab802c170 tx=0x7fbab802df90 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 msgr2=0x7fbac419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbab400e9f0 tx=0x7fbab400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 shutdown_connections 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbaac0a2910 0x7fbaac0a4dd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4105490 0x7fbac41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4104ad0 0x7fbac419a130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 --2- 172.21.15.123:0/479324208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac41038e0 0x7fbac4199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 >> 172.21.15.123:0/479324208 conn(0x7fbac40fee90 msgr2=0x7fbac4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 shutdown_connections 2024-03-24T00:35:41.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.297+0000 7fbacc800640 1 -- 172.21.15.123:0/479324208 wait complete. 2024-03-24T00:35:41.317 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-03-24T00:35:41.067491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630911,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":144,"num_read_kb":141,"num_write":238,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353748,"kb_used_data":12840,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588076,"statfs":{"total":1151940427776,"available":1151578189824,"internally_reserved":0,"allocated":13148160,"data_stored":7221539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40543,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":57,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.247663"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437381+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:35:34.437381+0000","last_peered":"2024-03-24T00:35:34.437381+0000","last_clean":"2024-03-24T00:35:34.437381+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:35:34.437381+0000","last_undegraded":"2024-03-24T00:35:34.437381+0000","last_fullsized":"2024-03-24T00:35:34.437381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415320+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:35:34.415320+0000","last_peered":"2024-03-24T00:35:34.415320+0000","last_clean":"2024-03-24T00:35:34.415320+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:35:34.415320+0000","last_undegraded":"2024-03-24T00:35:34.415320+0000","last_fullsized":"2024-03-24T00:35:34.415320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415741+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:35:34.415741+0000","last_peered":"2024-03-24T00:35:34.415741+0000","last_clean":"2024-03-24T00:35:34.415741+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:35:34.415741+0000","last_undegraded":"2024-03-24T00:35:34.415741+0000","last_fullsized":"2024-03-24T00:35:34.415741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412904+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:35:34.412904+0000","last_peered":"2024-03-24T00:35:34.412904+0000","last_clean":"2024-03-24T00:35:34.412904+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:35:34.412904+0000","last_undegraded":"2024-03-24T00:35:34.412904+0000","last_fullsized":"2024-03-24T00:35:34.412904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824931+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:35:33.824931+0000","last_peered":"2024-03-24T00:35:33.824931+0000","last_clean":"2024-03-24T00:35:33.824931+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:35:33.824931+0000","last_undegraded":"2024-03-24T00:35:33.824931+0000","last_fullsized":"2024-03-24T00:35:33.824931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413885+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:35:34.413885+0000","last_peered":"2024-03-24T00:35:34.413885+0000","last_clean":"2024-03-24T00:35:34.413885+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:35:34.413885+0000","last_undegraded":"2024-03-24T00:35:34.413885+0000","last_fullsized":"2024-03-24T00:35:34.413885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413530+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:35:34.413530+0000","last_peered":"2024-03-24T00:35:34.413530+0000","last_clean":"2024-03-24T00:35:34.413530+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:35:34.413530+0000","last_undegraded":"2024-03-24T00:35:34.413530+0000","last_fullsized":"2024-03-24T00:35:34.413530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361861+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:35:34.361861+0000","last_peered":"2024-03-24T00:35:34.361861+0000","last_clean":"2024-03-24T00:35:34.361861+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:35:34.361861+0000","last_undegraded":"2024-03-24T00:35:34.361861+0000","last_fullsized":"2024-03-24T00:35:34.361861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826202+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:35:33.826202+0000","last_peered":"2024-03-24T00:35:33.826202+0000","last_clean":"2024-03-24T00:35:33.826202+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:35:33.826202+0000","last_undegraded":"2024-03-24T00:35:33.826202+0000","last_fullsized":"2024-03-24T00:35:33.826202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414847+0000","last_change":"2024-03-24T00:35:32.837507+0000","last_active":"2024-03-24T00:35:34.414847+0000","last_peered":"2024-03-24T00:35:34.414847+0000","last_clean":"2024-03-24T00:35:34.414847+0000","last_became_active":"2024-03-24T00:35:32.837220+0000","last_became_peered":"2024-03-24T00:35:32.837220+0000","last_unstale":"2024-03-24T00:35:34.414847+0000","last_undegraded":"2024-03-24T00:35:34.414847+0000","last_fullsized":"2024-03-24T00:35:34.414847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:15.350491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414707+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:35:34.414707+0000","last_peered":"2024-03-24T00:35:34.414707+0000","last_clean":"2024-03-24T00:35:34.414707+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:35:34.414707+0000","last_undegraded":"2024-03-24T00:35:34.414707+0000","last_fullsized":"2024-03-24T00:35:34.414707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412850+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:35:34.412850+0000","last_peered":"2024-03-24T00:35:34.412850+0000","last_clean":"2024-03-24T00:35:34.412850+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:35:34.412850+0000","last_undegraded":"2024-03-24T00:35:34.412850+0000","last_fullsized":"2024-03-24T00:35:34.412850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827755+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:35:33.827755+0000","last_peered":"2024-03-24T00:35:33.827755+0000","last_clean":"2024-03-24T00:35:33.827755+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:35:33.827755+0000","last_undegraded":"2024-03-24T00:35:33.827755+0000","last_fullsized":"2024-03-24T00:35:33.827755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415144+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:35:34.415144+0000","last_peered":"2024-03-24T00:35:34.415144+0000","last_clean":"2024-03-24T00:35:34.415144+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:35:34.415144+0000","last_undegraded":"2024-03-24T00:35:34.415144+0000","last_fullsized":"2024-03-24T00:35:34.415144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414721+0000","last_change":"2024-03-24T00:35:32.852550+0000","last_active":"2024-03-24T00:35:34.414721+0000","last_peered":"2024-03-24T00:35:34.414721+0000","last_clean":"2024-03-24T00:35:34.414721+0000","last_became_active":"2024-03-24T00:35:32.848236+0000","last_became_peered":"2024-03-24T00:35:32.848236+0000","last_unstale":"2024-03-24T00:35:34.414721+0000","last_undegraded":"2024-03-24T00:35:34.414721+0000","last_fullsized":"2024-03-24T00:35:34.414721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:31.661283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825632+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:35:33.825632+0000","last_peered":"2024-03-24T00:35:33.825632+0000","last_clean":"2024-03-24T00:35:33.825632+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:35:33.825632+0000","last_undegraded":"2024-03-24T00:35:33.825632+0000","last_fullsized":"2024-03-24T00:35:33.825632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437491+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:35:34.437491+0000","last_peered":"2024-03-24T00:35:34.437491+0000","last_clean":"2024-03-24T00:35:34.437491+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:35:34.437491+0000","last_undegraded":"2024-03-24T00:35:34.437491+0000","last_fullsized":"2024-03-24T00:35:34.437491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063882+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:35:34.063882+0000","last_peered":"2024-03-24T00:35:34.063882+0000","last_clean":"2024-03-24T00:35:34.063882+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:35:34.063882+0000","last_undegraded":"2024-03-24T00:35:34.063882+0000","last_fullsized":"2024-03-24T00:35:34.063882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413631+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:35:34.413631+0000","last_peered":"2024-03-24T00:35:34.413631+0000","last_clean":"2024-03-24T00:35:34.413631+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:35:34.413631+0000","last_undegraded":"2024-03-24T00:35:34.413631+0000","last_fullsized":"2024-03-24T00:35:34.413631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063870+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:34.063870+0000","last_peered":"2024-03-24T00:35:34.063870+0000","last_clean":"2024-03-24T00:35:34.063870+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:34.063870+0000","last_undegraded":"2024-03-24T00:35:34.063870+0000","last_fullsized":"2024-03-24T00:35:34.063870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360867+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:35:34.360867+0000","last_peered":"2024-03-24T00:35:34.360867+0000","last_clean":"2024-03-24T00:35:34.360867+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:35:34.360867+0000","last_undegraded":"2024-03-24T00:35:34.360867+0000","last_fullsized":"2024-03-24T00:35:34.360867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361227+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:34.361227+0000","last_peered":"2024-03-24T00:35:34.361227+0000","last_clean":"2024-03-24T00:35:34.361227+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:34.361227+0000","last_undegraded":"2024-03-24T00:35:34.361227+0000","last_fullsized":"2024-03-24T00:35:34.361227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415878+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:35:34.415878+0000","last_peered":"2024-03-24T00:35:34.415878+0000","last_clean":"2024-03-24T00:35:34.415878+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:35:34.415878+0000","last_undegraded":"2024-03-24T00:35:34.415878+0000","last_fullsized":"2024-03-24T00:35:34.415878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825374+0000","last_change":"2024-03-24T00:35:32.337428+0000","last_active":"2024-03-24T00:35:33.825374+0000","last_peered":"2024-03-24T00:35:33.825374+0000","last_clean":"2024-03-24T00:35:33.825374+0000","last_became_active":"2024-03-24T00:35:32.337055+0000","last_became_peered":"2024-03-24T00:35:32.337055+0000","last_unstale":"2024-03-24T00:35:33.825374+0000","last_undegraded":"2024-03-24T00:35:33.825374+0000","last_fullsized":"2024-03-24T00:35:33.825374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:46.328889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825960+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:35:33.825960+0000","last_peered":"2024-03-24T00:35:33.825960+0000","last_clean":"2024-03-24T00:35:33.825960+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:35:33.825960+0000","last_undegraded":"2024-03-24T00:35:33.825960+0000","last_fullsized":"2024-03-24T00:35:33.825960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825122+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:33.825122+0000","last_peered":"2024-03-24T00:35:33.825122+0000","last_clean":"2024-03-24T00:35:33.825122+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:33.825122+0000","last_undegraded":"2024-03-24T00:35:33.825122+0000","last_fullsized":"2024-03-24T00:35:33.825122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327975+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:35:34.327975+0000","last_peered":"2024-03-24T00:35:34.327975+0000","last_clean":"2024-03-24T00:35:34.327975+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:35:34.327975+0000","last_undegraded":"2024-03-24T00:35:34.327975+0000","last_fullsized":"2024-03-24T00:35:34.327975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327657+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:34.327657+0000","last_peered":"2024-03-24T00:35:34.327657+0000","last_clean":"2024-03-24T00:35:34.327657+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:34.327657+0000","last_undegraded":"2024-03-24T00:35:34.327657+0000","last_fullsized":"2024-03-24T00:35:34.327657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824833+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:35:33.824833+0000","last_peered":"2024-03-24T00:35:33.824833+0000","last_clean":"2024-03-24T00:35:33.824833+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:35:33.824833+0000","last_undegraded":"2024-03-24T00:35:33.824833+0000","last_fullsized":"2024-03-24T00:35:33.824833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413723+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:34.413723+0000","last_peered":"2024-03-24T00:35:34.413723+0000","last_clean":"2024-03-24T00:35:34.413723+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:34.413723+0000","last_undegraded":"2024-03-24T00:35:34.413723+0000","last_fullsized":"2024-03-24T00:35:34.413723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327767+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:35:34.327767+0000","last_peered":"2024-03-24T00:35:34.327767+0000","last_clean":"2024-03-24T00:35:34.327767+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:35:34.327767+0000","last_undegraded":"2024-03-24T00:35:34.327767+0000","last_fullsized":"2024-03-24T00:35:34.327767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827920+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:33.827920+0000","last_peered":"2024-03-24T00:35:33.827920+0000","last_clean":"2024-03-24T00:35:33.827920+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:33.827920+0000","last_undegraded":"2024-03-24T00:35:33.827920+0000","last_fullsized":"2024-03-24T00:35:33.827920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360596+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:35:34.360596+0000","last_peered":"2024-03-24T00:35:34.360596+0000","last_clean":"2024-03-24T00:35:34.360596+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:35:34.360596+0000","last_undegraded":"2024-03-24T00:35:34.360596+0000","last_fullsized":"2024-03-24T00:35:34.360596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327409+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:34.327409+0000","last_peered":"2024-03-24T00:35:34.327409+0000","last_clean":"2024-03-24T00:35:34.327409+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:34.327409+0000","last_undegraded":"2024-03-24T00:35:34.327409+0000","last_fullsized":"2024-03-24T00:35:34.327409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063648+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:35:34.063648+0000","last_peered":"2024-03-24T00:35:34.063648+0000","last_clean":"2024-03-24T00:35:34.063648+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:35:34.063648+0000","last_undegraded":"2024-03-24T00:35:34.063648+0000","last_fullsized":"2024-03-24T00:35:34.063648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414810+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:34.414810+0000","last_peered":"2024-03-24T00:35:34.414810+0000","last_clean":"2024-03-24T00:35:34.414810+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:34.414810+0000","last_undegraded":"2024-03-24T00:35:34.414810+0000","last_fullsized":"2024-03-24T00:35:34.414810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826207+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:35:33.826207+0000","last_peered":"2024-03-24T00:35:33.826207+0000","last_clean":"2024-03-24T00:35:33.826207+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:35:33.826207+0000","last_undegraded":"2024-03-24T00:35:33.826207+0000","last_fullsized":"2024-03-24T00:35:33.826207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414219+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:34.414219+0000","last_peered":"2024-03-24T00:35:34.414219+0000","last_clean":"2024-03-24T00:35:34.414219+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:34.414219+0000","last_undegraded":"2024-03-24T00:35:34.414219+0000","last_fullsized":"2024-03-24T00:35:34.414219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415913+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:35:34.415913+0000","last_peered":"2024-03-24T00:35:34.415913+0000","last_clean":"2024-03-24T00:35:34.415913+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:35:34.415913+0000","last_undegraded":"2024-03-24T00:35:34.415913+0000","last_fullsized":"2024-03-24T00:35:34.415913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.328012+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:34.328012+0000","last_peered":"2024-03-24T00:35:34.328012+0000","last_clean":"2024-03-24T00:35:34.328012+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:34.328012+0000","last_undegraded":"2024-03-24T00:35:34.328012+0000","last_fullsized":"2024-03-24T00:35:34.328012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826114+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:35:33.826114+0000","last_peered":"2024-03-24T00:35:33.826114+0000","last_clean":"2024-03-24T00:35:33.826114+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:35:33.826114+0000","last_undegraded":"2024-03-24T00:35:33.826114+0000","last_fullsized":"2024-03-24T00:35:33.826114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'17","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:39.418513+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:35:39.418513+0000","last_peered":"2024-03-24T00:35:39.418513+0000","last_clean":"2024-03-24T00:35:39.418513+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:35:39.418513+0000","last_undegraded":"2024-03-24T00:35:39.418513+0000","last_fullsized":"2024-03-24T00:35:39.418513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39628,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":26,"num_write":17,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825238+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:35:33.825238+0000","last_peered":"2024-03-24T00:35:33.825238+0000","last_clean":"2024-03-24T00:35:33.825238+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:35:33.825238+0000","last_undegraded":"2024-03-24T00:35:33.825238+0000","last_fullsized":"2024-03-24T00:35:33.825238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:35.612647+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:35.612647+0000","last_peered":"2024-03-24T00:35:35.612647+0000","last_clean":"2024-03-24T00:35:35.612647+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:35.612647+0000","last_undegraded":"2024-03-24T00:35:35.612647+0000","last_fullsized":"2024-03-24T00:35:35.612647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.828058+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:35:33.828058+0000","last_peered":"2024-03-24T00:35:33.828058+0000","last_clean":"2024-03-24T00:35:33.828058+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:35:33.828058+0000","last_undegraded":"2024-03-24T00:35:33.828058+0000","last_fullsized":"2024-03-24T00:35:33.828058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327597+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:34.327597+0000","last_peered":"2024-03-24T00:35:34.327597+0000","last_clean":"2024-03-24T00:35:34.327597+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:34.327597+0000","last_undegraded":"2024-03-24T00:35:34.327597+0000","last_fullsized":"2024-03-24T00:35:34.327597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414431+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:35:34.414431+0000","last_peered":"2024-03-24T00:35:34.414431+0000","last_clean":"2024-03-24T00:35:34.414431+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:35:34.414431+0000","last_undegraded":"2024-03-24T00:35:34.414431+0000","last_fullsized":"2024-03-24T00:35:34.414431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412438+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:34.412438+0000","last_peered":"2024-03-24T00:35:34.412438+0000","last_clean":"2024-03-24T00:35:34.412438+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:34.412438+0000","last_undegraded":"2024-03-24T00:35:34.412438+0000","last_fullsized":"2024-03-24T00:35:34.412438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413934+0000","last_change":"2024-03-24T00:35:32.834033+0000","last_active":"2024-03-24T00:35:34.413934+0000","last_peered":"2024-03-24T00:35:34.413934+0000","last_clean":"2024-03-24T00:35:34.413934+0000","last_became_active":"2024-03-24T00:35:32.833701+0000","last_became_peered":"2024-03-24T00:35:32.833701+0000","last_unstale":"2024-03-24T00:35:34.413934+0000","last_undegraded":"2024-03-24T00:35:34.413934+0000","last_fullsized":"2024-03-24T00:35:34.413934+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:08.101266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824733+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:33.824733+0000","last_peered":"2024-03-24T00:35:33.824733+0000","last_clean":"2024-03-24T00:35:33.824733+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:33.824733+0000","last_undegraded":"2024-03-24T00:35:33.824733+0000","last_fullsized":"2024-03-24T00:35:33.824733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327622+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:35:34.327622+0000","last_peered":"2024-03-24T00:35:34.327622+0000","last_clean":"2024-03-24T00:35:34.327622+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:35:34.327622+0000","last_undegraded":"2024-03-24T00:35:34.327622+0000","last_fullsized":"2024-03-24T00:35:34.327622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.268259+0000","last_change":"2024-03-24T00:35:32.338542+0000","last_active":"2024-03-24T00:35:34.268259+0000","last_peered":"2024-03-24T00:35:34.268259+0000","last_clean":"2024-03-24T00:35:34.268259+0000","last_became_active":"2024-03-24T00:35:32.338235+0000","last_became_peered":"2024-03-24T00:35:32.338235+0000","last_unstale":"2024-03-24T00:35:34.268259+0000","last_undegraded":"2024-03-24T00:35:34.268259+0000","last_fullsized":"2024-03-24T00:35:34.268259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:38.248314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437284+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:35:34.437284+0000","last_peered":"2024-03-24T00:35:34.437284+0000","last_clean":"2024-03-24T00:35:34.437284+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:35:34.437284+0000","last_undegraded":"2024-03-24T00:35:34.437284+0000","last_fullsized":"2024-03-24T00:35:34.437284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063703+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:34.063703+0000","last_peered":"2024-03-24T00:35:34.063703+0000","last_clean":"2024-03-24T00:35:34.063703+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:34.063703+0000","last_undegraded":"2024-03-24T00:35:34.063703+0000","last_fullsized":"2024-03-24T00:35:34.063703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:35.048480+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:35:35.048480+0000","last_peered":"2024-03-24T00:35:35.048480+0000","last_clean":"2024-03-24T00:35:35.048480+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:35:35.048480+0000","last_undegraded":"2024-03-24T00:35:35.048480+0000","last_fullsized":"2024-03-24T00:35:35.048480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414205+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:34.414205+0000","last_peered":"2024-03-24T00:35:34.414205+0000","last_clean":"2024-03-24T00:35:34.414205+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:34.414205+0000","last_undegraded":"2024-03-24T00:35:34.414205+0000","last_fullsized":"2024-03-24T00:35:34.414205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414283+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:35:34.414283+0000","last_peered":"2024-03-24T00:35:34.414283+0000","last_clean":"2024-03-24T00:35:34.414283+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:35:34.414283+0000","last_undegraded":"2024-03-24T00:35:34.414283+0000","last_fullsized":"2024-03-24T00:35:34.414283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327702+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:35:34.327702+0000","last_peered":"2024-03-24T00:35:34.327702+0000","last_clean":"2024-03-24T00:35:34.327702+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:35:34.327702+0000","last_undegraded":"2024-03-24T00:35:34.327702+0000","last_fullsized":"2024-03-24T00:35:34.327702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826997+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:33.826997+0000","last_peered":"2024-03-24T00:35:33.826997+0000","last_clean":"2024-03-24T00:35:33.826997+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:33.826997+0000","last_undegraded":"2024-03-24T00:35:33.826997+0000","last_fullsized":"2024-03-24T00:35:33.826997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360615+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:35:34.360615+0000","last_peered":"2024-03-24T00:35:34.360615+0000","last_clean":"2024-03-24T00:35:34.360615+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:35:34.360615+0000","last_undegraded":"2024-03-24T00:35:34.360615+0000","last_fullsized":"2024-03-24T00:35:34.360615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412758+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:34.412758+0000","last_peered":"2024-03-24T00:35:34.412758+0000","last_clean":"2024-03-24T00:35:34.412758+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:34.412758+0000","last_undegraded":"2024-03-24T00:35:34.412758+0000","last_fullsized":"2024-03-24T00:35:34.412758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413669+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:34.413669+0000","last_peered":"2024-03-24T00:35:34.413669+0000","last_clean":"2024-03-24T00:35:34.413669+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:34.413669+0000","last_undegraded":"2024-03-24T00:35:34.413669+0000","last_fullsized":"2024-03-24T00:35:34.413669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827723+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:35:33.827723+0000","last_peered":"2024-03-24T00:35:33.827723+0000","last_clean":"2024-03-24T00:35:33.827723+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:35:33.827723+0000","last_undegraded":"2024-03-24T00:35:33.827723+0000","last_fullsized":"2024-03-24T00:35:33.827723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413544+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:34.413544+0000","last_peered":"2024-03-24T00:35:34.413544+0000","last_clean":"2024-03-24T00:35:34.413544+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:34.413544+0000","last_undegraded":"2024-03-24T00:35:34.413544+0000","last_fullsized":"2024-03-24T00:35:34.413544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825244+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:33.825244+0000","last_peered":"2024-03-24T00:35:33.825244+0000","last_clean":"2024-03-24T00:35:33.825244+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:33.825244+0000","last_undegraded":"2024-03-24T00:35:33.825244+0000","last_fullsized":"2024-03-24T00:35:33.825244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827983+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:35:33.827983+0000","last_peered":"2024-03-24T00:35:33.827983+0000","last_clean":"2024-03-24T00:35:33.827983+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:35:33.827983+0000","last_undegraded":"2024-03-24T00:35:33.827983+0000","last_fullsized":"2024-03-24T00:35:33.827983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063865+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:35:34.063865+0000","last_peered":"2024-03-24T00:35:34.063865+0000","last_clean":"2024-03-24T00:35:34.063865+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:35:34.063865+0000","last_undegraded":"2024-03-24T00:35:34.063865+0000","last_fullsized":"2024-03-24T00:35:34.063865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360964+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:35:34.360964+0000","last_peered":"2024-03-24T00:35:34.360964+0000","last_clean":"2024-03-24T00:35:34.360964+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:35:34.360964+0000","last_undegraded":"2024-03-24T00:35:34.360964+0000","last_fullsized":"2024-03-24T00:35:34.360964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412825+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:34.412825+0000","last_peered":"2024-03-24T00:35:34.412825+0000","last_clean":"2024-03-24T00:35:34.412825+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:34.412825+0000","last_undegraded":"2024-03-24T00:35:34.412825+0000","last_fullsized":"2024-03-24T00:35:34.412825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415749+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:35:34.415749+0000","last_peered":"2024-03-24T00:35:34.415749+0000","last_clean":"2024-03-24T00:35:34.415749+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:35:34.415749+0000","last_undegraded":"2024-03-24T00:35:34.415749+0000","last_fullsized":"2024-03-24T00:35:34.415749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825650+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:33.825650+0000","last_peered":"2024-03-24T00:35:33.825650+0000","last_clean":"2024-03-24T00:35:33.825650+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:33.825650+0000","last_undegraded":"2024-03-24T00:35:33.825650+0000","last_fullsized":"2024-03-24T00:35:33.825650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824850+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:35:33.824850+0000","last_peered":"2024-03-24T00:35:33.824850+0000","last_clean":"2024-03-24T00:35:33.824850+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:35:33.824850+0000","last_undegraded":"2024-03-24T00:35:33.824850+0000","last_fullsized":"2024-03-24T00:35:33.824850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360073+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:34.360073+0000","last_peered":"2024-03-24T00:35:34.360073+0000","last_clean":"2024-03-24T00:35:34.360073+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:34.360073+0000","last_undegraded":"2024-03-24T00:35:34.360073+0000","last_fullsized":"2024-03-24T00:35:34.360073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437183+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:35:34.437183+0000","last_peered":"2024-03-24T00:35:34.437183+0000","last_clean":"2024-03-24T00:35:34.437183+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:35:34.437183+0000","last_undegraded":"2024-03-24T00:35:34.437183+0000","last_fullsized":"2024-03-24T00:35:34.437183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413192+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:34.413192+0000","last_peered":"2024-03-24T00:35:34.413192+0000","last_clean":"2024-03-24T00:35:34.413192+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:34.413192+0000","last_undegraded":"2024-03-24T00:35:34.413192+0000","last_fullsized":"2024-03-24T00:35:34.413192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361064+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:35:34.361064+0000","last_peered":"2024-03-24T00:35:34.361064+0000","last_clean":"2024-03-24T00:35:34.361064+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:35:34.361064+0000","last_undegraded":"2024-03-24T00:35:34.361064+0000","last_fullsized":"2024-03-24T00:35:34.361064+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.268166+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:34.268166+0000","last_peered":"2024-03-24T00:35:34.268166+0000","last_clean":"2024-03-24T00:35:34.268166+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:34.268166+0000","last_undegraded":"2024-03-24T00:35:34.268166+0000","last_fullsized":"2024-03-24T00:35:34.268166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414178+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:35:34.414178+0000","last_peered":"2024-03-24T00:35:34.414178+0000","last_clean":"2024-03-24T00:35:34.414178+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:35:34.414178+0000","last_undegraded":"2024-03-24T00:35:34.414178+0000","last_fullsized":"2024-03-24T00:35:34.414178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413585+0000","last_change":"2024-03-24T00:35:32.355977+0000","last_active":"2024-03-24T00:35:34.413585+0000","last_peered":"2024-03-24T00:35:34.413585+0000","last_clean":"2024-03-24T00:35:34.413585+0000","last_became_active":"2024-03-24T00:35:32.355615+0000","last_became_peered":"2024-03-24T00:35:32.355615+0000","last_unstale":"2024-03-24T00:35:34.413585+0000","last_undegraded":"2024-03-24T00:35:34.413585+0000","last_fullsized":"2024-03-24T00:35:34.413585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:17.460377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414600+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:35:34.414600+0000","last_peered":"2024-03-24T00:35:34.414600+0000","last_clean":"2024-03-24T00:35:34.414600+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:35:34.414600+0000","last_undegraded":"2024-03-24T00:35:34.414600+0000","last_fullsized":"2024-03-24T00:35:34.414600+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:36.040802+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:36.040802+0000","last_peered":"2024-03-24T00:35:36.040802+0000","last_clean":"2024-03-24T00:35:36.040802+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:36.040802+0000","last_undegraded":"2024-03-24T00:35:36.040802+0000","last_fullsized":"2024-03-24T00:35:36.040802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414340+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:35:34.414340+0000","last_peered":"2024-03-24T00:35:34.414340+0000","last_clean":"2024-03-24T00:35:34.414340+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:35:34.414340+0000","last_undegraded":"2024-03-24T00:35:34.414340+0000","last_fullsized":"2024-03-24T00:35:34.414340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827917+0000","last_change":"2024-03-24T00:35:32.357910+0000","last_active":"2024-03-24T00:35:33.827917+0000","last_peered":"2024-03-24T00:35:33.827917+0000","last_clean":"2024-03-24T00:35:33.827917+0000","last_became_active":"2024-03-24T00:35:32.356325+0000","last_became_peered":"2024-03-24T00:35:32.356325+0000","last_unstale":"2024-03-24T00:35:33.827917+0000","last_undegraded":"2024-03-24T00:35:33.827917+0000","last_fullsized":"2024-03-24T00:35:33.827917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:44.698124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414537+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:35:34.414537+0000","last_peered":"2024-03-24T00:35:34.414537+0000","last_clean":"2024-03-24T00:35:34.414537+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:35:34.414537+0000","last_undegraded":"2024-03-24T00:35:34.414537+0000","last_fullsized":"2024-03-24T00:35:34.414537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437215+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:35:34.437215+0000","last_peered":"2024-03-24T00:35:34.437215+0000","last_clean":"2024-03-24T00:35:34.437215+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:35:34.437215+0000","last_undegraded":"2024-03-24T00:35:34.437215+0000","last_fullsized":"2024-03-24T00:35:34.437215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413541+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:35:34.413541+0000","last_peered":"2024-03-24T00:35:34.413541+0000","last_clean":"2024-03-24T00:35:34.413541+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:35:34.413541+0000","last_undegraded":"2024-03-24T00:35:34.413541+0000","last_fullsized":"2024-03-24T00:35:34.413541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413503+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:35:34.413503+0000","last_peered":"2024-03-24T00:35:34.413503+0000","last_clean":"2024-03-24T00:35:34.413503+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:35:34.413503+0000","last_undegraded":"2024-03-24T00:35:34.413503+0000","last_fullsized":"2024-03-24T00:35:34.413503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415056+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:35:34.415056+0000","last_peered":"2024-03-24T00:35:34.415056+0000","last_clean":"2024-03-24T00:35:34.415056+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:35:34.415056+0000","last_undegraded":"2024-03-24T00:35:34.415056+0000","last_fullsized":"2024-03-24T00:35:34.415056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.828031+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:35:33.828031+0000","last_peered":"2024-03-24T00:35:33.828031+0000","last_clean":"2024-03-24T00:35:33.828031+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:35:33.828031+0000","last_undegraded":"2024-03-24T00:35:33.828031+0000","last_fullsized":"2024-03-24T00:35:33.828031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437360+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:35:34.437360+0000","last_peered":"2024-03-24T00:35:34.437360+0000","last_clean":"2024-03-24T00:35:34.437360+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:35:34.437360+0000","last_undegraded":"2024-03-24T00:35:34.437360+0000","last_fullsized":"2024-03-24T00:35:34.437360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413842+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:35:34.413842+0000","last_peered":"2024-03-24T00:35:34.413842+0000","last_clean":"2024-03-24T00:35:34.413842+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:35:34.413842+0000","last_undegraded":"2024-03-24T00:35:34.413842+0000","last_fullsized":"2024-03-24T00:35:34.413842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360614+0000","last_change":"2024-03-24T00:35:32.864999+0000","last_active":"2024-03-24T00:35:34.360614+0000","last_peered":"2024-03-24T00:35:34.360614+0000","last_clean":"2024-03-24T00:35:34.360614+0000","last_became_active":"2024-03-24T00:35:32.859170+0000","last_became_peered":"2024-03-24T00:35:32.859170+0000","last_unstale":"2024-03-24T00:35:34.360614+0000","last_undegraded":"2024-03-24T00:35:34.360614+0000","last_fullsized":"2024-03-24T00:35:34.360614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:20:48.661164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327627+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:35:34.327627+0000","last_peered":"2024-03-24T00:35:34.327627+0000","last_clean":"2024-03-24T00:35:34.327627+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:35:34.327627+0000","last_undegraded":"2024-03-24T00:35:34.327627+0000","last_fullsized":"2024-03-24T00:35:34.327627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360421+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:35:34.360421+0000","last_peered":"2024-03-24T00:35:34.360421+0000","last_clean":"2024-03-24T00:35:34.360421+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:35:34.360421+0000","last_undegraded":"2024-03-24T00:35:34.360421+0000","last_fullsized":"2024-03-24T00:35:34.360421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825963+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:35:33.825963+0000","last_peered":"2024-03-24T00:35:33.825963+0000","last_clean":"2024-03-24T00:35:33.825963+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:35:33.825963+0000","last_undegraded":"2024-03-24T00:35:33.825963+0000","last_fullsized":"2024-03-24T00:35:33.825963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827838+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:35:33.827838+0000","last_peered":"2024-03-24T00:35:33.827838+0000","last_clean":"2024-03-24T00:35:33.827838+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:35:33.827838+0000","last_undegraded":"2024-03-24T00:35:33.827838+0000","last_fullsized":"2024-03-24T00:35:33.827838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437082+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:35:34.437082+0000","last_peered":"2024-03-24T00:35:34.437082+0000","last_clean":"2024-03-24T00:35:34.437082+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:35:34.437082+0000","last_undegraded":"2024-03-24T00:35:34.437082+0000","last_fullsized":"2024-03-24T00:35:34.437082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40426,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":56,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":83262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318288,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":933888,"data_stored":448806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579924,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708280,"statfs":{"total":95995035648,"available":95957278720,"internally_reserved":0,"allocated":1572864,"data_stored":1060380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776154,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":449388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":449410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1531904,"data_stored":1039756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462953,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":448923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":946176,"data_stored":450418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":448896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":488468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":448910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":448828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724591,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1540096,"data_stored":1039356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":39822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:35:41.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:35:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 -- 172.21.15.123:0/2111747910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e41038e0 msgr2=0x7f51e4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 --2- 172.21.15.123:0/2111747910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e41038e0 0x7f51e4103d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009900 tx=0x7f51d002bc50 comp rx=0 tx=0).stop 2024-03-24T00:35:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 -- 172.21.15.123:0/2111747910 shutdown_connections 2024-03-24T00:35:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 --2- 172.21.15.123:0/2111747910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 0x7f51e410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 --2- 172.21.15.123:0/2111747910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e4104ad0 0x7f51e4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 --2- 172.21.15.123:0/2111747910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e41038e0 0x7f51e4103d00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 -- 172.21.15.123:0/2111747910 >> 172.21.15.123:0/2111747910 conn(0x7f51e40fee90 msgr2=0x7f51e41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.574+0000 7f51ec400640 1 -- 172.21.15.123:0/2111747910 shutdown_connections 2024-03-24T00:35:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.575+0000 7f51ec400640 1 -- 172.21.15.123:0/2111747910 wait complete. 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 Processor -- start 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 -- start start 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 0x7f51e419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 0x7f51e41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e410f5b0 con 0x7f51e41038e0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e410f430 con 0x7f51e4104ad0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51ec400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e410f730 con 0x7f51e4105490 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33560/0 (socket says 172.21.15.123:33560) 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51e9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 0x7f51e419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.576+0000 7f51e9a00640 1 -- 172.21.15.123:0/2426900487 learned_addr learned my addr 172.21.15.123:0/2426900487 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51ea400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 0x7f51e41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 -- 172.21.15.123:0/2426900487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 msgr2=0x7f51e41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 0x7f51e41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 -- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 msgr2=0x7f51e4199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 -- 172.21.15.123:0/2426900487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e41a1810 con 0x7f51e4104ad0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9a00640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e9000640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 0x7f51e419a130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f51d800e9f0 tx=0x7f51d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d800ccc0 con 0x7f51e4104ad0 2024-03-24T00:35:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51d8010910 con 0x7f51e4104ad0 2024-03-24T00:35:41.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e41a1b00 con 0x7f51e4104ad0 2024-03-24T00:35:41.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8018960 con 0x7f51e4104ad0 2024-03-24T00:35:41.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.577+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e41a2050 con 0x7f51e4104ad0 2024-03-24T00:35:41.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.579+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e410aba0 con 0x7f51e4104ad0 2024-03-24T00:35:41.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.580+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f51d8010ac0 con 0x7f51e4104ad0 2024-03-24T00:35:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.582+0000 7f51e2000640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 0x7f51cc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.582+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51d80cf0a0 con 0x7f51e4104ad0 2024-03-24T00:35:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.582+0000 7f51e9a00640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 0x7f51cc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.582+0000 7f51e9a00640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 0x7f51cc0a4d50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f51d002c900 tx=0x7f51d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.590+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f51d809bf90 con 0x7f51e4104ad0 2024-03-24T00:35:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3485976671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-24T00:35:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: pgmap v317: 97 pgs: 97 active+clean; 616 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-03-24T00:35:41.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:41 smithi123 ceph-mon[31487]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3485976671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-24T00:35:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-24T00:35:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: pgmap v317: 97 pgs: 97 active+clean; 616 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-03-24T00:35:41.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:41 smithi177 ceph-mon[36034]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:41.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.787+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51e4061ae0 con 0x7f51cc0a2890 2024-03-24T00:35:41.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.810+0000 7f51e2000640 1 -- 172.21.15.123:0/2426900487 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258694 (secure 0 0 0) 0x7f51e4061ae0 con 0x7f51cc0a2890 2024-03-24T00:35:41.813 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.816+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 msgr2=0x7f51cc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.816+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 0x7f51cc0a4d50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f51d002c900 tx=0x7f51d0000bc0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 msgr2=0x7f51e419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 0x7f51e419a130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f51d800e9f0 tx=0x7f51d800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 shutdown_connections 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e4105490 0x7f51e41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51e4104ad0 0x7f51e419a130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51cc0a2890 0x7f51cc0a4d50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 --2- 172.21.15.123:0/2426900487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e41038e0 0x7f51e4199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 >> 172.21.15.123:0/2426900487 conn(0x7f51e40fee90 msgr2=0x7f51e4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 shutdown_connections 2024-03-24T00:35:41.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:41.817+0000 7f51ec400640 1 -- 172.21.15.123:0/2426900487 wait complete. 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3485976671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:35:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T00:35:41.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-24T00:35:41.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-24T00:35:41.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-24T00:35:41.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: pgmap v317: 97 pgs: 97 active+clean; 616 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-03-24T00:35:41.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:41 smithi204 ceph-mon[36345]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:41.837 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-03-24T00:35:41.419923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630911,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":144,"num_read_kb":141,"num_write":238,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353740,"kb_used_data":12844,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588084,"statfs":{"total":1151940427776,"available":1151578198016,"internally_reserved":0,"allocated":13152256,"data_stored":7221656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40543,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":57,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.355523"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437381+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:35:34.437381+0000","last_peered":"2024-03-24T00:35:34.437381+0000","last_clean":"2024-03-24T00:35:34.437381+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:35:34.437381+0000","last_undegraded":"2024-03-24T00:35:34.437381+0000","last_fullsized":"2024-03-24T00:35:34.437381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415320+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:35:34.415320+0000","last_peered":"2024-03-24T00:35:34.415320+0000","last_clean":"2024-03-24T00:35:34.415320+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:35:34.415320+0000","last_undegraded":"2024-03-24T00:35:34.415320+0000","last_fullsized":"2024-03-24T00:35:34.415320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415741+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:35:34.415741+0000","last_peered":"2024-03-24T00:35:34.415741+0000","last_clean":"2024-03-24T00:35:34.415741+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:35:34.415741+0000","last_undegraded":"2024-03-24T00:35:34.415741+0000","last_fullsized":"2024-03-24T00:35:34.415741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412904+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:35:34.412904+0000","last_peered":"2024-03-24T00:35:34.412904+0000","last_clean":"2024-03-24T00:35:34.412904+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:35:34.412904+0000","last_undegraded":"2024-03-24T00:35:34.412904+0000","last_fullsized":"2024-03-24T00:35:34.412904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824931+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:35:33.824931+0000","last_peered":"2024-03-24T00:35:33.824931+0000","last_clean":"2024-03-24T00:35:33.824931+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:35:33.824931+0000","last_undegraded":"2024-03-24T00:35:33.824931+0000","last_fullsized":"2024-03-24T00:35:33.824931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413885+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:35:34.413885+0000","last_peered":"2024-03-24T00:35:34.413885+0000","last_clean":"2024-03-24T00:35:34.413885+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:35:34.413885+0000","last_undegraded":"2024-03-24T00:35:34.413885+0000","last_fullsized":"2024-03-24T00:35:34.413885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413530+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:35:34.413530+0000","last_peered":"2024-03-24T00:35:34.413530+0000","last_clean":"2024-03-24T00:35:34.413530+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:35:34.413530+0000","last_undegraded":"2024-03-24T00:35:34.413530+0000","last_fullsized":"2024-03-24T00:35:34.413530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361861+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:35:34.361861+0000","last_peered":"2024-03-24T00:35:34.361861+0000","last_clean":"2024-03-24T00:35:34.361861+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:35:34.361861+0000","last_undegraded":"2024-03-24T00:35:34.361861+0000","last_fullsized":"2024-03-24T00:35:34.361861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826202+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:35:33.826202+0000","last_peered":"2024-03-24T00:35:33.826202+0000","last_clean":"2024-03-24T00:35:33.826202+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:35:33.826202+0000","last_undegraded":"2024-03-24T00:35:33.826202+0000","last_fullsized":"2024-03-24T00:35:33.826202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414847+0000","last_change":"2024-03-24T00:35:32.837507+0000","last_active":"2024-03-24T00:35:34.414847+0000","last_peered":"2024-03-24T00:35:34.414847+0000","last_clean":"2024-03-24T00:35:34.414847+0000","last_became_active":"2024-03-24T00:35:32.837220+0000","last_became_peered":"2024-03-24T00:35:32.837220+0000","last_unstale":"2024-03-24T00:35:34.414847+0000","last_undegraded":"2024-03-24T00:35:34.414847+0000","last_fullsized":"2024-03-24T00:35:34.414847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:15.350491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414707+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:35:34.414707+0000","last_peered":"2024-03-24T00:35:34.414707+0000","last_clean":"2024-03-24T00:35:34.414707+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:35:34.414707+0000","last_undegraded":"2024-03-24T00:35:34.414707+0000","last_fullsized":"2024-03-24T00:35:34.414707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412850+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:35:34.412850+0000","last_peered":"2024-03-24T00:35:34.412850+0000","last_clean":"2024-03-24T00:35:34.412850+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:35:34.412850+0000","last_undegraded":"2024-03-24T00:35:34.412850+0000","last_fullsized":"2024-03-24T00:35:34.412850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827755+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:35:33.827755+0000","last_peered":"2024-03-24T00:35:33.827755+0000","last_clean":"2024-03-24T00:35:33.827755+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:35:33.827755+0000","last_undegraded":"2024-03-24T00:35:33.827755+0000","last_fullsized":"2024-03-24T00:35:33.827755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415144+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:35:34.415144+0000","last_peered":"2024-03-24T00:35:34.415144+0000","last_clean":"2024-03-24T00:35:34.415144+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:35:34.415144+0000","last_undegraded":"2024-03-24T00:35:34.415144+0000","last_fullsized":"2024-03-24T00:35:34.415144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414721+0000","last_change":"2024-03-24T00:35:32.852550+0000","last_active":"2024-03-24T00:35:34.414721+0000","last_peered":"2024-03-24T00:35:34.414721+0000","last_clean":"2024-03-24T00:35:34.414721+0000","last_became_active":"2024-03-24T00:35:32.848236+0000","last_became_peered":"2024-03-24T00:35:32.848236+0000","last_unstale":"2024-03-24T00:35:34.414721+0000","last_undegraded":"2024-03-24T00:35:34.414721+0000","last_fullsized":"2024-03-24T00:35:34.414721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:31.661283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825632+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:35:33.825632+0000","last_peered":"2024-03-24T00:35:33.825632+0000","last_clean":"2024-03-24T00:35:33.825632+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:35:33.825632+0000","last_undegraded":"2024-03-24T00:35:33.825632+0000","last_fullsized":"2024-03-24T00:35:33.825632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437491+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:35:34.437491+0000","last_peered":"2024-03-24T00:35:34.437491+0000","last_clean":"2024-03-24T00:35:34.437491+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:35:34.437491+0000","last_undegraded":"2024-03-24T00:35:34.437491+0000","last_fullsized":"2024-03-24T00:35:34.437491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063882+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:35:34.063882+0000","last_peered":"2024-03-24T00:35:34.063882+0000","last_clean":"2024-03-24T00:35:34.063882+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:35:34.063882+0000","last_undegraded":"2024-03-24T00:35:34.063882+0000","last_fullsized":"2024-03-24T00:35:34.063882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413631+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:35:34.413631+0000","last_peered":"2024-03-24T00:35:34.413631+0000","last_clean":"2024-03-24T00:35:34.413631+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:35:34.413631+0000","last_undegraded":"2024-03-24T00:35:34.413631+0000","last_fullsized":"2024-03-24T00:35:34.413631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063870+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:34.063870+0000","last_peered":"2024-03-24T00:35:34.063870+0000","last_clean":"2024-03-24T00:35:34.063870+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:34.063870+0000","last_undegraded":"2024-03-24T00:35:34.063870+0000","last_fullsized":"2024-03-24T00:35:34.063870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360867+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:35:34.360867+0000","last_peered":"2024-03-24T00:35:34.360867+0000","last_clean":"2024-03-24T00:35:34.360867+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:35:34.360867+0000","last_undegraded":"2024-03-24T00:35:34.360867+0000","last_fullsized":"2024-03-24T00:35:34.360867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361227+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:34.361227+0000","last_peered":"2024-03-24T00:35:34.361227+0000","last_clean":"2024-03-24T00:35:34.361227+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:34.361227+0000","last_undegraded":"2024-03-24T00:35:34.361227+0000","last_fullsized":"2024-03-24T00:35:34.361227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415878+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:35:34.415878+0000","last_peered":"2024-03-24T00:35:34.415878+0000","last_clean":"2024-03-24T00:35:34.415878+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:35:34.415878+0000","last_undegraded":"2024-03-24T00:35:34.415878+0000","last_fullsized":"2024-03-24T00:35:34.415878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825374+0000","last_change":"2024-03-24T00:35:32.337428+0000","last_active":"2024-03-24T00:35:33.825374+0000","last_peered":"2024-03-24T00:35:33.825374+0000","last_clean":"2024-03-24T00:35:33.825374+0000","last_became_active":"2024-03-24T00:35:32.337055+0000","last_became_peered":"2024-03-24T00:35:32.337055+0000","last_unstale":"2024-03-24T00:35:33.825374+0000","last_undegraded":"2024-03-24T00:35:33.825374+0000","last_fullsized":"2024-03-24T00:35:33.825374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:46.328889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825960+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:35:33.825960+0000","last_peered":"2024-03-24T00:35:33.825960+0000","last_clean":"2024-03-24T00:35:33.825960+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:35:33.825960+0000","last_undegraded":"2024-03-24T00:35:33.825960+0000","last_fullsized":"2024-03-24T00:35:33.825960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825122+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:33.825122+0000","last_peered":"2024-03-24T00:35:33.825122+0000","last_clean":"2024-03-24T00:35:33.825122+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:33.825122+0000","last_undegraded":"2024-03-24T00:35:33.825122+0000","last_fullsized":"2024-03-24T00:35:33.825122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327975+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:35:34.327975+0000","last_peered":"2024-03-24T00:35:34.327975+0000","last_clean":"2024-03-24T00:35:34.327975+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:35:34.327975+0000","last_undegraded":"2024-03-24T00:35:34.327975+0000","last_fullsized":"2024-03-24T00:35:34.327975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327657+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:34.327657+0000","last_peered":"2024-03-24T00:35:34.327657+0000","last_clean":"2024-03-24T00:35:34.327657+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:34.327657+0000","last_undegraded":"2024-03-24T00:35:34.327657+0000","last_fullsized":"2024-03-24T00:35:34.327657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824833+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:35:33.824833+0000","last_peered":"2024-03-24T00:35:33.824833+0000","last_clean":"2024-03-24T00:35:33.824833+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:35:33.824833+0000","last_undegraded":"2024-03-24T00:35:33.824833+0000","last_fullsized":"2024-03-24T00:35:33.824833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413723+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:34.413723+0000","last_peered":"2024-03-24T00:35:34.413723+0000","last_clean":"2024-03-24T00:35:34.413723+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:34.413723+0000","last_undegraded":"2024-03-24T00:35:34.413723+0000","last_fullsized":"2024-03-24T00:35:34.413723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327767+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:35:34.327767+0000","last_peered":"2024-03-24T00:35:34.327767+0000","last_clean":"2024-03-24T00:35:34.327767+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:35:34.327767+0000","last_undegraded":"2024-03-24T00:35:34.327767+0000","last_fullsized":"2024-03-24T00:35:34.327767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827920+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:33.827920+0000","last_peered":"2024-03-24T00:35:33.827920+0000","last_clean":"2024-03-24T00:35:33.827920+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:33.827920+0000","last_undegraded":"2024-03-24T00:35:33.827920+0000","last_fullsized":"2024-03-24T00:35:33.827920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360596+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:35:34.360596+0000","last_peered":"2024-03-24T00:35:34.360596+0000","last_clean":"2024-03-24T00:35:34.360596+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:35:34.360596+0000","last_undegraded":"2024-03-24T00:35:34.360596+0000","last_fullsized":"2024-03-24T00:35:34.360596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327409+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:34.327409+0000","last_peered":"2024-03-24T00:35:34.327409+0000","last_clean":"2024-03-24T00:35:34.327409+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:34.327409+0000","last_undegraded":"2024-03-24T00:35:34.327409+0000","last_fullsized":"2024-03-24T00:35:34.327409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063648+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:35:34.063648+0000","last_peered":"2024-03-24T00:35:34.063648+0000","last_clean":"2024-03-24T00:35:34.063648+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:35:34.063648+0000","last_undegraded":"2024-03-24T00:35:34.063648+0000","last_fullsized":"2024-03-24T00:35:34.063648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414810+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:34.414810+0000","last_peered":"2024-03-24T00:35:34.414810+0000","last_clean":"2024-03-24T00:35:34.414810+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:34.414810+0000","last_undegraded":"2024-03-24T00:35:34.414810+0000","last_fullsized":"2024-03-24T00:35:34.414810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826207+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:35:33.826207+0000","last_peered":"2024-03-24T00:35:33.826207+0000","last_clean":"2024-03-24T00:35:33.826207+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:35:33.826207+0000","last_undegraded":"2024-03-24T00:35:33.826207+0000","last_fullsized":"2024-03-24T00:35:33.826207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414219+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:34.414219+0000","last_peered":"2024-03-24T00:35:34.414219+0000","last_clean":"2024-03-24T00:35:34.414219+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:34.414219+0000","last_undegraded":"2024-03-24T00:35:34.414219+0000","last_fullsized":"2024-03-24T00:35:34.414219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415913+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:35:34.415913+0000","last_peered":"2024-03-24T00:35:34.415913+0000","last_clean":"2024-03-24T00:35:34.415913+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:35:34.415913+0000","last_undegraded":"2024-03-24T00:35:34.415913+0000","last_fullsized":"2024-03-24T00:35:34.415913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.328012+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:34.328012+0000","last_peered":"2024-03-24T00:35:34.328012+0000","last_clean":"2024-03-24T00:35:34.328012+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:34.328012+0000","last_undegraded":"2024-03-24T00:35:34.328012+0000","last_fullsized":"2024-03-24T00:35:34.328012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826114+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:35:33.826114+0000","last_peered":"2024-03-24T00:35:33.826114+0000","last_clean":"2024-03-24T00:35:33.826114+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:35:33.826114+0000","last_undegraded":"2024-03-24T00:35:33.826114+0000","last_fullsized":"2024-03-24T00:35:33.826114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'17","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:39.418513+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:35:39.418513+0000","last_peered":"2024-03-24T00:35:39.418513+0000","last_clean":"2024-03-24T00:35:39.418513+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:35:39.418513+0000","last_undegraded":"2024-03-24T00:35:39.418513+0000","last_fullsized":"2024-03-24T00:35:39.418513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39628,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":26,"num_write":17,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825238+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:35:33.825238+0000","last_peered":"2024-03-24T00:35:33.825238+0000","last_clean":"2024-03-24T00:35:33.825238+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:35:33.825238+0000","last_undegraded":"2024-03-24T00:35:33.825238+0000","last_fullsized":"2024-03-24T00:35:33.825238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:35.612647+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:35.612647+0000","last_peered":"2024-03-24T00:35:35.612647+0000","last_clean":"2024-03-24T00:35:35.612647+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:35.612647+0000","last_undegraded":"2024-03-24T00:35:35.612647+0000","last_fullsized":"2024-03-24T00:35:35.612647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.828058+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:35:33.828058+0000","last_peered":"2024-03-24T00:35:33.828058+0000","last_clean":"2024-03-24T00:35:33.828058+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:35:33.828058+0000","last_undegraded":"2024-03-24T00:35:33.828058+0000","last_fullsized":"2024-03-24T00:35:33.828058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327597+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:34.327597+0000","last_peered":"2024-03-24T00:35:34.327597+0000","last_clean":"2024-03-24T00:35:34.327597+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:34.327597+0000","last_undegraded":"2024-03-24T00:35:34.327597+0000","last_fullsized":"2024-03-24T00:35:34.327597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414431+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:35:34.414431+0000","last_peered":"2024-03-24T00:35:34.414431+0000","last_clean":"2024-03-24T00:35:34.414431+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:35:34.414431+0000","last_undegraded":"2024-03-24T00:35:34.414431+0000","last_fullsized":"2024-03-24T00:35:34.414431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412438+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:34.412438+0000","last_peered":"2024-03-24T00:35:34.412438+0000","last_clean":"2024-03-24T00:35:34.412438+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:34.412438+0000","last_undegraded":"2024-03-24T00:35:34.412438+0000","last_fullsized":"2024-03-24T00:35:34.412438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413934+0000","last_change":"2024-03-24T00:35:32.834033+0000","last_active":"2024-03-24T00:35:34.413934+0000","last_peered":"2024-03-24T00:35:34.413934+0000","last_clean":"2024-03-24T00:35:34.413934+0000","last_became_active":"2024-03-24T00:35:32.833701+0000","last_became_peered":"2024-03-24T00:35:32.833701+0000","last_unstale":"2024-03-24T00:35:34.413934+0000","last_undegraded":"2024-03-24T00:35:34.413934+0000","last_fullsized":"2024-03-24T00:35:34.413934+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:08.101266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824733+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:33.824733+0000","last_peered":"2024-03-24T00:35:33.824733+0000","last_clean":"2024-03-24T00:35:33.824733+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:33.824733+0000","last_undegraded":"2024-03-24T00:35:33.824733+0000","last_fullsized":"2024-03-24T00:35:33.824733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327622+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:35:34.327622+0000","last_peered":"2024-03-24T00:35:34.327622+0000","last_clean":"2024-03-24T00:35:34.327622+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:35:34.327622+0000","last_undegraded":"2024-03-24T00:35:34.327622+0000","last_fullsized":"2024-03-24T00:35:34.327622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.268259+0000","last_change":"2024-03-24T00:35:32.338542+0000","last_active":"2024-03-24T00:35:34.268259+0000","last_peered":"2024-03-24T00:35:34.268259+0000","last_clean":"2024-03-24T00:35:34.268259+0000","last_became_active":"2024-03-24T00:35:32.338235+0000","last_became_peered":"2024-03-24T00:35:32.338235+0000","last_unstale":"2024-03-24T00:35:34.268259+0000","last_undegraded":"2024-03-24T00:35:34.268259+0000","last_fullsized":"2024-03-24T00:35:34.268259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:38.248314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437284+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:35:34.437284+0000","last_peered":"2024-03-24T00:35:34.437284+0000","last_clean":"2024-03-24T00:35:34.437284+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:35:34.437284+0000","last_undegraded":"2024-03-24T00:35:34.437284+0000","last_fullsized":"2024-03-24T00:35:34.437284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063703+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:34.063703+0000","last_peered":"2024-03-24T00:35:34.063703+0000","last_clean":"2024-03-24T00:35:34.063703+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:34.063703+0000","last_undegraded":"2024-03-24T00:35:34.063703+0000","last_fullsized":"2024-03-24T00:35:34.063703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:35.048480+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:35:35.048480+0000","last_peered":"2024-03-24T00:35:35.048480+0000","last_clean":"2024-03-24T00:35:35.048480+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:35:35.048480+0000","last_undegraded":"2024-03-24T00:35:35.048480+0000","last_fullsized":"2024-03-24T00:35:35.048480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414205+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:34.414205+0000","last_peered":"2024-03-24T00:35:34.414205+0000","last_clean":"2024-03-24T00:35:34.414205+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:34.414205+0000","last_undegraded":"2024-03-24T00:35:34.414205+0000","last_fullsized":"2024-03-24T00:35:34.414205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414283+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:35:34.414283+0000","last_peered":"2024-03-24T00:35:34.414283+0000","last_clean":"2024-03-24T00:35:34.414283+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:35:34.414283+0000","last_undegraded":"2024-03-24T00:35:34.414283+0000","last_fullsized":"2024-03-24T00:35:34.414283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327702+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:35:34.327702+0000","last_peered":"2024-03-24T00:35:34.327702+0000","last_clean":"2024-03-24T00:35:34.327702+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:35:34.327702+0000","last_undegraded":"2024-03-24T00:35:34.327702+0000","last_fullsized":"2024-03-24T00:35:34.327702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.826997+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:33.826997+0000","last_peered":"2024-03-24T00:35:33.826997+0000","last_clean":"2024-03-24T00:35:33.826997+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:33.826997+0000","last_undegraded":"2024-03-24T00:35:33.826997+0000","last_fullsized":"2024-03-24T00:35:33.826997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360615+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:35:34.360615+0000","last_peered":"2024-03-24T00:35:34.360615+0000","last_clean":"2024-03-24T00:35:34.360615+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:35:34.360615+0000","last_undegraded":"2024-03-24T00:35:34.360615+0000","last_fullsized":"2024-03-24T00:35:34.360615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412758+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:34.412758+0000","last_peered":"2024-03-24T00:35:34.412758+0000","last_clean":"2024-03-24T00:35:34.412758+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:34.412758+0000","last_undegraded":"2024-03-24T00:35:34.412758+0000","last_fullsized":"2024-03-24T00:35:34.412758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413669+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:34.413669+0000","last_peered":"2024-03-24T00:35:34.413669+0000","last_clean":"2024-03-24T00:35:34.413669+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:34.413669+0000","last_undegraded":"2024-03-24T00:35:34.413669+0000","last_fullsized":"2024-03-24T00:35:34.413669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827723+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:35:33.827723+0000","last_peered":"2024-03-24T00:35:33.827723+0000","last_clean":"2024-03-24T00:35:33.827723+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:35:33.827723+0000","last_undegraded":"2024-03-24T00:35:33.827723+0000","last_fullsized":"2024-03-24T00:35:33.827723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413544+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:34.413544+0000","last_peered":"2024-03-24T00:35:34.413544+0000","last_clean":"2024-03-24T00:35:34.413544+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:34.413544+0000","last_undegraded":"2024-03-24T00:35:34.413544+0000","last_fullsized":"2024-03-24T00:35:34.413544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825244+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:33.825244+0000","last_peered":"2024-03-24T00:35:33.825244+0000","last_clean":"2024-03-24T00:35:33.825244+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:33.825244+0000","last_undegraded":"2024-03-24T00:35:33.825244+0000","last_fullsized":"2024-03-24T00:35:33.825244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827983+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:35:33.827983+0000","last_peered":"2024-03-24T00:35:33.827983+0000","last_clean":"2024-03-24T00:35:33.827983+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:35:33.827983+0000","last_undegraded":"2024-03-24T00:35:33.827983+0000","last_fullsized":"2024-03-24T00:35:33.827983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.063865+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:35:34.063865+0000","last_peered":"2024-03-24T00:35:34.063865+0000","last_clean":"2024-03-24T00:35:34.063865+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:35:34.063865+0000","last_undegraded":"2024-03-24T00:35:34.063865+0000","last_fullsized":"2024-03-24T00:35:34.063865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360964+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:35:34.360964+0000","last_peered":"2024-03-24T00:35:34.360964+0000","last_clean":"2024-03-24T00:35:34.360964+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:35:34.360964+0000","last_undegraded":"2024-03-24T00:35:34.360964+0000","last_fullsized":"2024-03-24T00:35:34.360964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.412825+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:34.412825+0000","last_peered":"2024-03-24T00:35:34.412825+0000","last_clean":"2024-03-24T00:35:34.412825+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:34.412825+0000","last_undegraded":"2024-03-24T00:35:34.412825+0000","last_fullsized":"2024-03-24T00:35:34.412825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415749+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:35:34.415749+0000","last_peered":"2024-03-24T00:35:34.415749+0000","last_clean":"2024-03-24T00:35:34.415749+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:35:34.415749+0000","last_undegraded":"2024-03-24T00:35:34.415749+0000","last_fullsized":"2024-03-24T00:35:34.415749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825650+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:33.825650+0000","last_peered":"2024-03-24T00:35:33.825650+0000","last_clean":"2024-03-24T00:35:33.825650+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:33.825650+0000","last_undegraded":"2024-03-24T00:35:33.825650+0000","last_fullsized":"2024-03-24T00:35:33.825650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.824850+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:35:33.824850+0000","last_peered":"2024-03-24T00:35:33.824850+0000","last_clean":"2024-03-24T00:35:33.824850+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:35:33.824850+0000","last_undegraded":"2024-03-24T00:35:33.824850+0000","last_fullsized":"2024-03-24T00:35:33.824850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360073+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:34.360073+0000","last_peered":"2024-03-24T00:35:34.360073+0000","last_clean":"2024-03-24T00:35:34.360073+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:34.360073+0000","last_undegraded":"2024-03-24T00:35:34.360073+0000","last_fullsized":"2024-03-24T00:35:34.360073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437183+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:35:34.437183+0000","last_peered":"2024-03-24T00:35:34.437183+0000","last_clean":"2024-03-24T00:35:34.437183+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:35:34.437183+0000","last_undegraded":"2024-03-24T00:35:34.437183+0000","last_fullsized":"2024-03-24T00:35:34.437183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413192+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:34.413192+0000","last_peered":"2024-03-24T00:35:34.413192+0000","last_clean":"2024-03-24T00:35:34.413192+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:34.413192+0000","last_undegraded":"2024-03-24T00:35:34.413192+0000","last_fullsized":"2024-03-24T00:35:34.413192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.361064+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:35:34.361064+0000","last_peered":"2024-03-24T00:35:34.361064+0000","last_clean":"2024-03-24T00:35:34.361064+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:35:34.361064+0000","last_undegraded":"2024-03-24T00:35:34.361064+0000","last_fullsized":"2024-03-24T00:35:34.361064+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.268166+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:34.268166+0000","last_peered":"2024-03-24T00:35:34.268166+0000","last_clean":"2024-03-24T00:35:34.268166+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:34.268166+0000","last_undegraded":"2024-03-24T00:35:34.268166+0000","last_fullsized":"2024-03-24T00:35:34.268166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414178+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:35:34.414178+0000","last_peered":"2024-03-24T00:35:34.414178+0000","last_clean":"2024-03-24T00:35:34.414178+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:35:34.414178+0000","last_undegraded":"2024-03-24T00:35:34.414178+0000","last_fullsized":"2024-03-24T00:35:34.414178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413585+0000","last_change":"2024-03-24T00:35:32.355977+0000","last_active":"2024-03-24T00:35:34.413585+0000","last_peered":"2024-03-24T00:35:34.413585+0000","last_clean":"2024-03-24T00:35:34.413585+0000","last_became_active":"2024-03-24T00:35:32.355615+0000","last_became_peered":"2024-03-24T00:35:32.355615+0000","last_unstale":"2024-03-24T00:35:34.413585+0000","last_undegraded":"2024-03-24T00:35:34.413585+0000","last_fullsized":"2024-03-24T00:35:34.413585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:17.460377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414600+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:35:34.414600+0000","last_peered":"2024-03-24T00:35:34.414600+0000","last_clean":"2024-03-24T00:35:34.414600+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:35:34.414600+0000","last_undegraded":"2024-03-24T00:35:34.414600+0000","last_fullsized":"2024-03-24T00:35:34.414600+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:36.040802+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:36.040802+0000","last_peered":"2024-03-24T00:35:36.040802+0000","last_clean":"2024-03-24T00:35:36.040802+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:36.040802+0000","last_undegraded":"2024-03-24T00:35:36.040802+0000","last_fullsized":"2024-03-24T00:35:36.040802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414340+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:35:34.414340+0000","last_peered":"2024-03-24T00:35:34.414340+0000","last_clean":"2024-03-24T00:35:34.414340+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:35:34.414340+0000","last_undegraded":"2024-03-24T00:35:34.414340+0000","last_fullsized":"2024-03-24T00:35:34.414340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827917+0000","last_change":"2024-03-24T00:35:32.357910+0000","last_active":"2024-03-24T00:35:33.827917+0000","last_peered":"2024-03-24T00:35:33.827917+0000","last_clean":"2024-03-24T00:35:33.827917+0000","last_became_active":"2024-03-24T00:35:32.356325+0000","last_became_peered":"2024-03-24T00:35:32.356325+0000","last_unstale":"2024-03-24T00:35:33.827917+0000","last_undegraded":"2024-03-24T00:35:33.827917+0000","last_fullsized":"2024-03-24T00:35:33.827917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:44.698124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.414537+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:35:34.414537+0000","last_peered":"2024-03-24T00:35:34.414537+0000","last_clean":"2024-03-24T00:35:34.414537+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:35:34.414537+0000","last_undegraded":"2024-03-24T00:35:34.414537+0000","last_fullsized":"2024-03-24T00:35:34.414537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437215+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:35:34.437215+0000","last_peered":"2024-03-24T00:35:34.437215+0000","last_clean":"2024-03-24T00:35:34.437215+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:35:34.437215+0000","last_undegraded":"2024-03-24T00:35:34.437215+0000","last_fullsized":"2024-03-24T00:35:34.437215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413541+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:35:34.413541+0000","last_peered":"2024-03-24T00:35:34.413541+0000","last_clean":"2024-03-24T00:35:34.413541+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:35:34.413541+0000","last_undegraded":"2024-03-24T00:35:34.413541+0000","last_fullsized":"2024-03-24T00:35:34.413541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413503+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:35:34.413503+0000","last_peered":"2024-03-24T00:35:34.413503+0000","last_clean":"2024-03-24T00:35:34.413503+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:35:34.413503+0000","last_undegraded":"2024-03-24T00:35:34.413503+0000","last_fullsized":"2024-03-24T00:35:34.413503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.415056+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:35:34.415056+0000","last_peered":"2024-03-24T00:35:34.415056+0000","last_clean":"2024-03-24T00:35:34.415056+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:35:34.415056+0000","last_undegraded":"2024-03-24T00:35:34.415056+0000","last_fullsized":"2024-03-24T00:35:34.415056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.828031+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:35:33.828031+0000","last_peered":"2024-03-24T00:35:33.828031+0000","last_clean":"2024-03-24T00:35:33.828031+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:35:33.828031+0000","last_undegraded":"2024-03-24T00:35:33.828031+0000","last_fullsized":"2024-03-24T00:35:33.828031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437360+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:35:34.437360+0000","last_peered":"2024-03-24T00:35:34.437360+0000","last_clean":"2024-03-24T00:35:34.437360+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:35:34.437360+0000","last_undegraded":"2024-03-24T00:35:34.437360+0000","last_fullsized":"2024-03-24T00:35:34.437360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.413842+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:35:34.413842+0000","last_peered":"2024-03-24T00:35:34.413842+0000","last_clean":"2024-03-24T00:35:34.413842+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:35:34.413842+0000","last_undegraded":"2024-03-24T00:35:34.413842+0000","last_fullsized":"2024-03-24T00:35:34.413842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360614+0000","last_change":"2024-03-24T00:35:32.864999+0000","last_active":"2024-03-24T00:35:34.360614+0000","last_peered":"2024-03-24T00:35:34.360614+0000","last_clean":"2024-03-24T00:35:34.360614+0000","last_became_active":"2024-03-24T00:35:32.859170+0000","last_became_peered":"2024-03-24T00:35:32.859170+0000","last_unstale":"2024-03-24T00:35:34.360614+0000","last_undegraded":"2024-03-24T00:35:34.360614+0000","last_fullsized":"2024-03-24T00:35:34.360614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:20:48.661164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.327627+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:35:34.327627+0000","last_peered":"2024-03-24T00:35:34.327627+0000","last_clean":"2024-03-24T00:35:34.327627+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:35:34.327627+0000","last_undegraded":"2024-03-24T00:35:34.327627+0000","last_fullsized":"2024-03-24T00:35:34.327627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.360421+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:35:34.360421+0000","last_peered":"2024-03-24T00:35:34.360421+0000","last_clean":"2024-03-24T00:35:34.360421+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:35:34.360421+0000","last_undegraded":"2024-03-24T00:35:34.360421+0000","last_fullsized":"2024-03-24T00:35:34.360421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.825963+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:35:33.825963+0000","last_peered":"2024-03-24T00:35:33.825963+0000","last_clean":"2024-03-24T00:35:33.825963+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:35:33.825963+0000","last_undegraded":"2024-03-24T00:35:33.825963+0000","last_fullsized":"2024-03-24T00:35:33.825963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:33.827838+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:35:33.827838+0000","last_peered":"2024-03-24T00:35:33.827838+0000","last_clean":"2024-03-24T00:35:33.827838+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:35:33.827838+0000","last_undegraded":"2024-03-24T00:35:33.827838+0000","last_fullsized":"2024-03-24T00:35:33.827838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T00:35:34.437082+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:35:34.437082+0000","last_peered":"2024-03-24T00:35:34.437082+0000","last_clean":"2024-03-24T00:35:34.437082+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:35:34.437082+0000","last_undegraded":"2024-03-24T00:35:34.437082+0000","last_fullsized":"2024-03-24T00:35:34.437082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40426,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":56,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":83262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318288,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":933888,"data_stored":448806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579924,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708280,"statfs":{"total":95995035648,"available":95957278720,"internally_reserved":0,"allocated":1572864,"data_stored":1060380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776154,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":449388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":449410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1531904,"data_stored":1039756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462953,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":448923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":946176,"data_stored":450418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":448896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":488468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":448910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":448828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1039473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":39822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:35:41.843 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:35:41.843 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T00:35:41.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T00:35:42.326 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-24T00:35:42.688 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-24T00:35:42.689 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:35:42.689 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:42 smithi123 ceph-mon[31487]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T00:35:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-24T00:35:42.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-24T00:35:42.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-24T00:35:42.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:35:42.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:42 smithi177 ceph-mon[36034]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-24T00:35:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-24T00:35:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T00:35:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:35:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T00:35:42.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-24T00:35:42.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-24T00:35:42.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-24T00:35:42.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:35:42.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:42 smithi204 ceph-mon[36345]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:35:43.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:43 smithi123 ceph-mon[31487]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:35:43.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:43 smithi123 ceph-mon[31487]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 616 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-03-24T00:35:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:43 smithi177 ceph-mon[36034]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:35:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:43 smithi177 ceph-mon[36034]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 616 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-03-24T00:35:43.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:43 smithi204 ceph-mon[36345]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:35:43.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:43 smithi204 ceph-mon[36345]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 616 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-03-24T00:35:44.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 -- 172.21.15.123:0/2140528972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 msgr2=0x7f1064107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 --2- 172.21.15.123:0/2140528972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064107990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1050009990 tx=0x7f1050009ca0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 -- 172.21.15.123:0/2140528972 shutdown_connections 2024-03-24T00:35:44.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 --2- 172.21.15.123:0/2140528972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 0x7f106410a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 --2- 172.21.15.123:0/2140528972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f1064105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 --2- 172.21.15.123:0/2140528972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064107990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.541+0000 7f1068e00700 1 -- 172.21.15.123:0/2140528972 >> 172.21.15.123:0/2140528972 conn(0x7f10640fc680 msgr2=0x7f10640feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.542+0000 7f1068e00700 1 -- 172.21.15.123:0/2140528972 shutdown_connections 2024-03-24T00:35:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.542+0000 7f1068e00700 1 -- 172.21.15.123:0/2140528972 wait complete. 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.543+0000 7f1068e00700 1 Processor -- start 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.543+0000 7f1068e00700 1 -- start start 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f10641025d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 0x7f1064101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410de40 con 0x7f1064102c70 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410dcc0 con 0x7f10641055a0 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1068e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410dfc0 con 0x7f1064107ed0 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f10641025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46758/0 (socket says 172.21.15.123:46758) 2024-03-24T00:35:44.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 -- 172.21.15.123:0/3249749116 learned_addr learned my addr 172.21.15.123:0/3249749116 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:44.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1062800700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 0x7f1064101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 -- 172.21.15.123:0/3249749116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 msgr2=0x7f1064101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 0x7f1064101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 -- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 msgr2=0x7f10641025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f10641025d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061400700 1 -- 172.21.15.123:0/3249749116 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064101a10 con 0x7f10641055a0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.544+0000 7f1061e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f10641025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f1061400700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1050005910 tx=0x7f105000bb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050026070 con 0x7f10641055a0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1050003d70 con 0x7f10641055a0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050021670 con 0x7f10641055a0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10641a2460 con 0x7f10641055a0 2024-03-24T00:35:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.545+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641a28b0 con 0x7f10641055a0 2024-03-24T00:35:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.546+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10640506a0 con 0x7f10641055a0 2024-03-24T00:35:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.548+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1050029070 con 0x7f10641055a0 2024-03-24T00:35:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.549+0000 7f105aa00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 0x7f10480a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.549+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f10500cf9a0 con 0x7f10641055a0 2024-03-24T00:35:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.549+0000 7f1061e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 0x7f10480a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.550+0000 7f1061e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 0x7f10480a4f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1054009680 tx=0x7f1054009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.553+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f105005d740 con 0x7f10641055a0 2024-03-24T00:35:44.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.861+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f106402da30 con 0x7f10641055a0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.867+0000 7f105aa00700 1 -- 172.21.15.123:0/3249749116 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1050013070 con 0x7f10641055a0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBgdf9lJhyBMxAA9GWn5jhGylgZVa7GUv4cow== 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.869+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 msgr2=0x7f10480a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.869+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 0x7f10480a4f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1054009680 tx=0x7f1054009260 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 msgr2=0x7f1064100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1050005910 tx=0x7f105000bb90 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 shutdown_connections 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10480a2a50 0x7f10480a4f10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1064107ed0 0x7f1064101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1064102c70 0x7f10641025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 --2- 172.21.15.123:0/3249749116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10641055a0 0x7f1064100c20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 >> 172.21.15.123:0/3249749116 conn(0x7f10640fc680 msgr2=0x7f10641035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 shutdown_connections 2024-03-24T00:35:44.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:44.870+0000 7f1068e00700 1 -- 172.21.15.123:0/3249749116 wait complete. 2024-03-24T00:35:45.189 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:44 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3249749116' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.189 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:44 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.189 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:44 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:35:45.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:44 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3249749116' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:44 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:44 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:35:45.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:44 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3249749116' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:44 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:35:45.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:44 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:35:45.711 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:35:45.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T00:35:45.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T00:35:45.850 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T00:35:45.850 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T00:35:45.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T00:35:46.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:45 smithi177 ceph-mon[36034]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:46.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:45 smithi204 ceph-mon[36345]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:46.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:45 smithi123 ceph-mon[31487]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:48.347 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:48 smithi123 ceph-mon[31487]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:48.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:48 smithi177 ceph-mon[36034]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 -- 172.21.15.123:0/3687567070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 msgr2=0x7fbefc108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 --2- 172.21.15.123:0/3687567070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc108c20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009990 tx=0x7fbee8009ca0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 -- 172.21.15.123:0/3687567070 shutdown_connections 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 --2- 172.21.15.123:0/3687567070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc109160 0x7fbefc115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 --2- 172.21.15.123:0/3687567070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc10e7a0 0x7fbefc10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 --2- 172.21.15.123:0/3687567070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc108c20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 -- 172.21.15.123:0/3687567070 >> 172.21.15.123:0/3687567070 conn(0x7fbefc075960 msgr2=0x7fbefc077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 -- 172.21.15.123:0/3687567070 shutdown_connections 2024-03-24T00:35:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.510+0000 7fbf00a00700 1 -- 172.21.15.123:0/3687567070 wait complete. 2024-03-24T00:35:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.511+0000 7fbf00a00700 1 Processor -- start 2024-03-24T00:35:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.511+0000 7fbf00a00700 1 -- start start 2024-03-24T00:35:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.511+0000 7fbf00a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbf00a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 0x7fbefc19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbf00a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc10e7a0 0x7fbefc1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbf00a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc118b80 con 0x7fbefc1087a0 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbf00a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc118a00 con 0x7fbefc10e7a0 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbf00a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc118d00 con 0x7fbefc109160 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 0x7fbefc19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57900/0 (socket says 172.21.15.123:57900) 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9e00700 1 -- 172.21.15.123:0/4137377593 learned_addr learned my addr 172.21.15.123:0/4137377593 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbefa800700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc10e7a0 0x7fbefc1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 -- 172.21.15.123:0/4137377593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc10e7a0 msgr2=0x7fbefc1a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc10e7a0 0x7fbefc1a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 -- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 msgr2=0x7fbefc19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.512+0000 7fbef9400700 1 -- 172.21.15.123:0/4137377593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc1a3fb0 con 0x7fbefc109160 2024-03-24T00:35:48.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbef9400700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 0x7fbefc19f530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000c00 tx=0x7fbee800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8026070 con 0x7fbefc109160 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc1a4240 con 0x7fbefc109160 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc1abb20 con 0x7fbefc109160 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbee8003d70 con 0x7fbefc109160 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.513+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8021410 con 0x7fbefc109160 2024-03-24T00:35:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.514+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc005330 con 0x7fbefc109160 2024-03-24T00:35:48.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.516+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbee8029070 con 0x7fbefc109160 2024-03-24T00:35:48.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.517+0000 7fbef2a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 0x7fbee00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.517+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbee80cf7e0 con 0x7fbefc109160 2024-03-24T00:35:48.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.517+0000 7fbef9e00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 0x7fbee00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.518+0000 7fbef9e00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 0x7fbee00a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbeec009ca0 tx=0x7fbeec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:48.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.522+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbee805d740 con 0x7fbefc109160 2024-03-24T00:35:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:48 smithi204 ceph-mon[36345]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.803+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbedc0058f0 con 0x7fbefc109160 2024-03-24T00:35:48.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.810+0000 7fbef2a00700 1 -- 172.21.15.123:0/4137377593 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262667 (secure 0 0 0) 0x7fbee8013070 con 0x7fbefc109160 2024-03-24T00:35:48.817 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 msgr2=0x7fbee00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 0x7fbee00a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbeec009ca0 tx=0x7fbeec009260 comp rx=0 tx=0).stop 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 msgr2=0x7fbefc19f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 0x7fbefc19f530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000c00 tx=0x7fbee800bab0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 shutdown_connections 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbee00a2920 0x7fbee00a4de0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbefc10e7a0 0x7fbefc1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc1087a0 0x7fbefc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 --2- 172.21.15.123:0/4137377593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbefc109160 0x7fbefc19f530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 >> 172.21.15.123:0/4137377593 conn(0x7fbefc075960 msgr2=0x7fbefc076040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:48.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 shutdown_connections 2024-03-24T00:35:48.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:48.816+0000 7fbf00a00700 1 -- 172.21.15.123:0/4137377593 wait complete. 2024-03-24T00:35:49.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:49 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4137377593' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:35:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:49 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4137377593' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:35:49.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:49 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4137377593' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:35:49.678 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2933502234},{"type":"v1","addr":"172.21.15.123:6801","nonce":2933502234}]},"active_addr":"172.21.15.123:6801/2933502234","active_change":"2024-03-24T00:27:40.925387+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.123:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":3644708448}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":126021721}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":2581123664}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":197627345}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":272596799}]}]} 2024-03-24T00:35:49.687 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T00:35:49.687 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T00:35:49.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:35:50.135 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:50.415 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:50 smithi123 ceph-mon[31487]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:50.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:50 smithi177 ceph-mon[36034]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:50 smithi204 ceph-mon[36345]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:52.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 -- 172.21.15.123:0/2446247667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac06a760 msgr2=0x7f68ac10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 --2- 172.21.15.123:0/2446247667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac06a760 0x7f68ac10e700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6898009990 tx=0x7f6898009ca0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 -- 172.21.15.123:0/2446247667 shutdown_connections 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 --2- 172.21.15.123:0/2446247667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac10ec40 0x7f68ac111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 --2- 172.21.15.123:0/2446247667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 0x7f68ac06a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 --2- 172.21.15.123:0/2446247667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac06a760 0x7f68ac10e700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 -- 172.21.15.123:0/2446247667 >> 172.21.15.123:0/2446247667 conn(0x7f68ac077fc0 msgr2=0x7f68ac0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 -- 172.21.15.123:0/2446247667 shutdown_connections 2024-03-24T00:35:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.343+0000 7f68b4200700 1 -- 172.21.15.123:0/2446247667 wait complete. 2024-03-24T00:35:52.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.344+0000 7f68b4200700 1 Processor -- start 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.344+0000 7f68b4200700 1 -- start start 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 0x7f68ac1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 0x7f68ac19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1146c0 con 0x7f68ac10ec40 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac114540 con 0x7f68ac06a760 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b4200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac114840 con 0x7f68ac069e40 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39046/0 (socket says 172.21.15.123:39046) 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b0e00700 1 -- 172.21.15.123:0/2798484938 learned_addr learned my addr 172.21.15.123:0/2798484938 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b1800700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 0x7f68ac1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.345+0000 7f68b2200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 0x7f68ac19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 -- 172.21.15.123:0/2798484938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 msgr2=0x7f68ac1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 0x7f68ac1a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 -- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 msgr2=0x7f68ac19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 0x7f68ac19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 -- 172.21.15.123:0/2798484938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac19b160 con 0x7f68ac06a760 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b2200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 0x7f68ac19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:52.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b0e00700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6898005910 tx=0x7f689800bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898027070 con 0x7f68ac06a760 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6898003d70 con 0x7f68ac06a760 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac19b3f0 con 0x7f68ac06a760 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.346+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800f6c0 con 0x7f68ac06a760 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.347+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1a7710 con 0x7f68ac06a760 2024-03-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.347+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac068e50 con 0x7f68ac06a760 2024-03-24T00:35:52.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.349+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f689802a070 con 0x7f68ac06a760 2024-03-24T00:35:52.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.350+0000 7f68aa000700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 0x7f68940a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.351+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f68980cfcb0 con 0x7f68ac06a760 2024-03-24T00:35:52.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.351+0000 7f68b1800700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 0x7f68940a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.351+0000 7f68b1800700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 0x7f68940a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f68ac19bb40 tx=0x7f68a000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.354+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f689805e740 con 0x7f68ac06a760 2024-03-24T00:35:52.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:52 smithi177 ceph-mon[36034]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:52 smithi204 ceph-mon[36345]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.575+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68ac02da30 con 0x7f68ac06a760 2024-03-24T00:35:52.576 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:52 smithi123 ceph-mon[31487]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.577+0000 7f68aa000700 1 -- 172.21.15.123:0/2798484938 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f6898013070 con 0x7f68ac06a760 2024-03-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 msgr2=0x7f68940a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 0x7f68940a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f68ac19bb40 tx=0x7f68a000b040 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 msgr2=0x7f68ac1a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6898005910 tx=0x7f689800bb60 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 shutdown_connections 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68940a29a0 0x7f68940a4e60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ac10ec40 0x7f68ac19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac069e40 0x7f68ac1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 --2- 172.21.15.123:0/2798484938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68ac06a760 0x7f68ac1a0d80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 >> 172.21.15.123:0/2798484938 conn(0x7f68ac077fc0 msgr2=0x7f68ac1100f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 shutdown_connections 2024-03-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:52.581+0000 7f68b4200700 1 -- 172.21.15.123:0/2798484938 wait complete. 2024-03-24T00:35:53.417 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2798484938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:53.421 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T00:35:53.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:35:53.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2798484938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2798484938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:53.880 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:54.326 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:54 smithi123 ceph-mon[31487]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:54.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:54 smithi177 ceph-mon[36034]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:54 smithi204 ceph-mon[36345]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-24T00:35:55.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 -- 172.21.15.123:0/1992904119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c06a7e0 msgr2=0x7f033c10f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 --2- 172.21.15.123:0/1992904119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c06a7e0 0x7f033c10f020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f033400b0a0 tx=0x7f033400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 -- 172.21.15.123:0/1992904119 shutdown_connections 2024-03-24T00:35:55.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 --2- 172.21.15.123:0/1992904119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c06a7e0 0x7f033c10f020 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 --2- 172.21.15.123:0/1992904119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c1081e0 0x7f033c1085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 --2- 172.21.15.123:0/1992904119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 -- 172.21.15.123:0/1992904119 >> 172.21.15.123:0/1992904119 conn(0x7f033c077fc0 msgr2=0x7f033c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 -- 172.21.15.123:0/1992904119 shutdown_connections 2024-03-24T00:35:55.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.932+0000 7f0343e00700 1 -- 172.21.15.123:0/1992904119 wait complete. 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 Processor -- start 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 -- start start 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c06a7e0 0x7f033c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c112480 con 0x7f033c069e40 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c112300 con 0x7f033c1081e0 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.933+0000 7f0343e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c112600 con 0x7f033c06a7e0 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45694/0 (socket says 172.21.15.123:45694) 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36574/0 (socket says 172.21.15.123:36574) 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 -- 172.21.15.123:0/3672969099 learned_addr learned my addr 172.21.15.123:0/3672969099 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0340a00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c06a7e0 0x7f033c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 -- 172.21.15.123:0/3672969099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c06a7e0 msgr2=0x7f033c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c06a7e0 0x7f033c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 -- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 msgr2=0x7f033c19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 -- 172.21.15.123:0/3672969099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c198fa0 con 0x7f033c1081e0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.934+0000 7f0341e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0334006070 tx=0x7f0334004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.935+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334021070 con 0x7f033c1081e0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.935+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0334003c00 con 0x7f033c1081e0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.935+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340047c0 con 0x7f033c1081e0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.935+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c199230 con 0x7f033c1081e0 2024-03-24T00:35:55.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.935+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c101090 con 0x7f033c1081e0 2024-03-24T00:35:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.936+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c103890 con 0x7f033c1081e0 2024-03-24T00:35:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.938+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0334007980 con 0x7f033c1081e0 2024-03-24T00:35:55.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.939+0000 7f033a000700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 0x7f03240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.939+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f03340cfb00 con 0x7f033c1081e0 2024-03-24T00:35:55.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.939+0000 7f0341400700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 0x7f03240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.940+0000 7f0341400700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 0x7f03240a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f033000ffb0 tx=0x7f033000f7a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:55.943+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f033409ca80 con 0x7f033c1081e0 2024-03-24T00:35:56.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.169+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f033c199c20 con 0x7f033c1081e0 2024-03-24T00:35:56.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.173+0000 7f033a000700 1 -- 172.21.15.123:0/3672969099 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f033c199c20 con 0x7f033c1081e0 2024-03-24T00:35:56.173 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:35:56.174 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:56.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.175+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 msgr2=0x7f03240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:56.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.175+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 0x7f03240a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f033000ffb0 tx=0x7f033000f7a0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.175+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 msgr2=0x7f033c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.175+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0334006070 tx=0x7f0334004cf0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 shutdown_connections 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f03240a29a0 0x7f03240a4e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033c1081e0 0x7f033c1986c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f033c069e40 0x7f033c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 --2- 172.21.15.123:0/3672969099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f033c06a7e0 0x7f033c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 >> 172.21.15.123:0/3672969099 conn(0x7f033c077fc0 msgr2=0x7f033c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:56.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 shutdown_connections 2024-03-24T00:35:56.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:35:56.176+0000 7f0343e00700 1 -- 172.21.15.123:0/3672969099 wait complete. 2024-03-24T00:35:56.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:56 smithi177 ceph-mon[36034]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-03-24T00:35:56.488 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:56 smithi123 ceph-mon[31487]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-03-24T00:35:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:56 smithi204 ceph-mon[36345]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-03-24T00:35:56.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T00:35:56.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T00:35:56.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T00:35:56.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T00:35:56.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T00:35:56.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T00:35:56.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T00:35:56.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T00:35:56.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T00:35:56.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T00:35:56.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T00:35:56.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T00:35:57.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:57 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3672969099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:57.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:57 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3672969099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:57 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3672969099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:35:57.943 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:57.960 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:58.079 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:35:58.408 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:35:58 smithi123 ceph-mon[31487]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:35:58.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:35:58 smithi177 ceph-mon[36034]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:35:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:35:58 smithi204 ceph-mon[36345]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:36:00.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:00 smithi177 ceph-mon[36034]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:36:00.513 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:00 smithi123 ceph-mon[31487]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:36:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:00 smithi204 ceph-mon[36345]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-24T00:36:02.105 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:02.107 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:02.108 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:02.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:02 smithi123 ceph-mon[31487]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-24T00:36:02.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:02 smithi177 ceph-mon[36034]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-24T00:36:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:02 smithi204 ceph-mon[36345]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 -- 172.21.15.123:0/2189406951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd41023d0 msgr2=0x7f8fd41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/2189406951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd41023d0 0x7f8fd41027b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009990 tx=0x7f8fc8009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 -- 172.21.15.123:0/2189406951 shutdown_connections 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/2189406951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fd41074b0 0x7f8fd410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/2189406951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd41023d0 0x7f8fd41027b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/2189406951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fd4102d80 0x7f8fd4106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 -- 172.21.15.123:0/2189406951 >> 172.21.15.123:0/2189406951 conn(0x7f8fd40fdc40 msgr2=0x7f8fd4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 -- 172.21.15.123:0/2189406951 shutdown_connections 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.850+0000 7f8fdbe00700 1 -- 172.21.15.123:0/2189406951 wait complete. 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.851+0000 7f8fdbe00700 1 Processor -- start 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 -- start start 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fd41023d0 0x7f8fd41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 0x7f8fd41a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fd41074b0 0x7f8fd419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41124e0 con 0x7f8fd41074b0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4112360 con 0x7f8fd4102d80 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fdbe00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4112660 con 0x7f8fd41023d0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fd8a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 0x7f8fd41a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fd8a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 0x7f8fd41a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45712/0 (socket says 172.21.15.123:45712) 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fd8a00700 1 -- 172.21.15.123:0/627753487 learned_addr learned my addr 172.21.15.123:0/627753487 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fd9400700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fd41023d0 0x7f8fd41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.852+0000 7f8fd9e00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fd41074b0 0x7f8fd419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 -- 172.21.15.123:0/627753487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fd41023d0 msgr2=0x7f8fd41a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fd41023d0 0x7f8fd41a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 -- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fd41074b0 msgr2=0x7f8fd419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fd41074b0 0x7f8fd419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd419b130 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd8a00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 0x7f8fd41a0d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000d980 tx=0x7f8fc000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0017070 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fc000fd70 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc000e3b0 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd419b420 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.853+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41a76e0 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.854+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8fd40756a0 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.856+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8fc000f890 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.857+0000 7f8fd2000700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8fbc0a29a0 0x7f8fbc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8fc00ce630 con 0x7f8fd4102d80 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd9400700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8fbc0a29a0 0x7f8fbc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd2000700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] conn(0x7f8fbc0ac4e0 0x7f8fbc0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fc0014ec0 con 0x7f8fbc0ac4e0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fc00d5050 con 0x7f8fd4102d80 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.858+0000 7f8fd9e00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] conn(0x7f8fbc0ac4e0 0x7f8fbc0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.859+0000 7f8fd9400700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8fbc0a29a0 0x7f8fbc0a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fd419bb10 tx=0x7f8fc800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.859+0000 7f8fd9e00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] conn(0x7f8fbc0ac4e0 0x7f8fbc0ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.860+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== osd.6 v2:172.21.15.123:6818/3042863338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8fc0014ec0 con 0x7f8fbc0ac4e0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.878+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 --> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8fd41038b0 con 0x7f8fbc0ac4e0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fd2000700 1 -- 172.21.15.123:0/627753487 <== osd.6 v2:172.21.15.123:6818/3042863338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8fd41038b0 con 0x7f8fbc0ac4e0 2024-03-24T00:36:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] conn(0x7f8fbc0ac4e0 msgr2=0x7f8fbc0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] conn(0x7f8fbc0ac4e0 0x7f8fbc0ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8fbc0a29a0 msgr2=0x7f8fbc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8fbc0a29a0 0x7f8fbc0a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fd419bb10 tx=0x7f8fc800b260 comp rx=0 tx=0).stop 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 msgr2=0x7f8fd41a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fdbe00700 1 --2- 172.21.15.123:0/627753487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd4102d80 0x7f8fd41a0d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000d980 tx=0x7f8fc000dc90 comp rx=0 tx=0).stop 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.879+0000 7f8fd9e00700 1 -- 172.21.15.123:0/627753487 reap_dead start 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.880+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 shutdown_connections 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.880+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 >> 172.21.15.123:0/627753487 conn(0x7f8fd40fdc40 msgr2=0x7f8fd40feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.880+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 shutdown_connections 2024-03-24T00:36:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:02.880+0000 7f8fdbe00700 1 -- 172.21.15.123:0/627753487 wait complete. 2024-03-24T00:36:04.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:04 smithi177 ceph-mon[36034]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:04.553 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:04 smithi204 ceph-mon[36345]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:04 smithi123 ceph-mon[31487]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 -- 172.21.15.123:0/2908530766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 msgr2=0x7f31541156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 --2- 172.21.15.123:0/2908530766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f31541156d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f315000b0a0 tx=0x7f315000b3b0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 -- 172.21.15.123:0/2908530766 shutdown_connections 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 --2- 172.21.15.123:0/2908530766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f31541156d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 --2- 172.21.15.123:0/2908530766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315410e800 0x7f315410ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 --2- 172.21.15.123:0/2908530766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154108800 0x7f3154108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 -- 172.21.15.123:0/2908530766 >> 172.21.15.123:0/2908530766 conn(0x7f31540767e0 msgr2=0x7f3154078c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 -- 172.21.15.123:0/2908530766 shutdown_connections 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.950+0000 7f315c800700 1 -- 172.21.15.123:0/2908530766 wait complete. 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.951+0000 7f315c800700 1 Processor -- start 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 -- start start 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154108800 0x7f315419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f315419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315410e800 0x7f31541a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3154118be0 con 0x7f3154108800 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3154118a60 con 0x7f31541091c0 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315c800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3154118d60 con 0x7f315410e800 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f3159e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154108800 0x7f315419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f3159400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f315419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f3159400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f315419f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45726/0 (socket says 172.21.15.123:45726) 2024-03-24T00:36:05.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f3159400700 1 -- 172.21.15.123:0/1142071943 learned_addr learned my addr 172.21.15.123:0/1142071943 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.952+0000 7f315a800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315410e800 0x7f31541a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 -- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315410e800 msgr2=0x7f31541a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315410e800 0x7f31541a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 -- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154108800 msgr2=0x7f315419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154108800 0x7f315419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31541a3fb0 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f3159400700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f315419f530 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f314400e9a0 tx=0x7f314400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314400cbd0 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f314400cd70 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144018750 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31541a42a0 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.953+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31541abb20 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.954+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31540628a0 con 0x7f31541091c0 2024-03-24T00:36:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.956+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3144010430 con 0x7f31541091c0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.957+0000 7f314ea00700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f313c0a29a0 0x7f313c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f31440ceff0 con 0x7f31541091c0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f3159e00700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f313c0a29a0 0x7f313c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f314ea00700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] conn(0x7f313c0ac4e0 0x7f313c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f313c0af050 con 0x7f313c0ac4e0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31440cf400 con 0x7f31541091c0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f3159e00700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f313c0a29a0 0x7f313c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3148009be0 tx=0x7f3148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.958+0000 7f315a800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] conn(0x7f313c0ac4e0 0x7f313c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.959+0000 7f315a800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] conn(0x7f313c0ac4e0 0x7f313c0ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.960+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== osd.11 v2:172.21.15.204:6824/2066593187 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f313c0af050 con 0x7f313c0ac4e0 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.979+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 --> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3154062ca0 con 0x7f313c0ac4e0 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.980+0000 7f314ea00700 1 -- 172.21.15.123:0/1142071943 <== osd.11 v2:172.21.15.204:6824/2066593187 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3154062ca0 con 0x7f313c0ac4e0 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] conn(0x7f313c0ac4e0 msgr2=0x7f313c0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] conn(0x7f313c0ac4e0 0x7f313c0ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f313c0a29a0 msgr2=0x7f313c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f313c0a29a0 0x7f313c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3148009be0 tx=0x7f3148009260 comp rx=0 tx=0).stop 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 msgr2=0x7f315419f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 --2- 172.21.15.123:0/1142071943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31541091c0 0x7f315419f530 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f314400e9a0 tx=0x7f314400ecb0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315a800700 1 -- 172.21.15.123:0/1142071943 reap_dead start 2024-03-24T00:36:05.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 shutdown_connections 2024-03-24T00:36:05.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 >> 172.21.15.123:0/1142071943 conn(0x7f31540767e0 msgr2=0x7f3154076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 shutdown_connections 2024-03-24T00:36:05.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:04.981+0000 7f315c800700 1 -- 172.21.15.123:0/1142071943 wait complete. 2024-03-24T00:36:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.869+0000 7f43b7c00700 1 -- 172.21.15.123:0/583195861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 msgr2=0x7f43b010eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.869+0000 7f43b7c00700 1 --2- 172.21.15.123:0/583195861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b010eb80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009990 tx=0x7f43a4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 -- 172.21.15.123:0/583195861 shutdown_connections 2024-03-24T00:36:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 --2- 172.21.15.123:0/583195861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b0109160 0x7f43b0115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 --2- 172.21.15.123:0/583195861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b010eb80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 --2- 172.21.15.123:0/583195861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b01087a0 0x7f43b0108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 -- 172.21.15.123:0/583195861 >> 172.21.15.123:0/583195861 conn(0x7f43b0075960 msgr2=0x7f43b0077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 -- 172.21.15.123:0/583195861 shutdown_connections 2024-03-24T00:36:05.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.870+0000 7f43b7c00700 1 -- 172.21.15.123:0/583195861 wait complete. 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.871+0000 7f43b7c00700 1 Processor -- start 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.871+0000 7f43b7c00700 1 -- start start 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 0x7f43b019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b0109160 0x7f43b019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b01a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0118b20 con 0x7f43b01087a0 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b01189a0 con 0x7f43b010e7a0 2024-03-24T00:36:05.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b7c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0118ca0 con 0x7f43b0109160 2024-03-24T00:36:05.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b5200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 0x7f43b019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b5c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b01a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b5c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b01a3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47666/0 (socket says 172.21.15.123:47666) 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b5200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 0x7f43b019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33574/0 (socket says 172.21.15.123:33574) 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43b5c00700 1 -- 172.21.15.123:0/1402880028 learned_addr learned my addr 172.21.15.123:0/1402880028 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.872+0000 7f43afe00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b0109160 0x7f43b019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 -- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b0109160 msgr2=0x7f43b019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b0109160 0x7f43b019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 -- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 msgr2=0x7f43b019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 0x7f43b019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b01a3f40 con 0x7f43b010e7a0 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43afe00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43b0109160 0x7f43b019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5200700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43b01087a0 0x7f43b019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b5c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b01a3800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43a800ef30 tx=0x7f43a800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a80092e0 con 0x7f43b010e7a0 2024-03-24T00:36:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43a8004bb0 con 0x7f43b010e7a0 2024-03-24T00:36:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b01a41d0 con 0x7f43b010e7a0 2024-03-24T00:36:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8004280 con 0x7f43b010e7a0 2024-03-24T00:36:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.873+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b01abab0 con 0x7f43b010e7a0 2024-03-24T00:36:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.874+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f43b0062920 con 0x7f43b010e7a0 2024-03-24T00:36:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.876+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f43a801b030 con 0x7f43b010e7a0 2024-03-24T00:36:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.877+0000 7f43ad600700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43980a29a0 0x7f43980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.878+0000 7f43b5200700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43980a29a0 0x7f43980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.878+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f43a80ce650 con 0x7f43b010e7a0 2024-03-24T00:36:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.878+0000 7f43ad600700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] conn(0x7f43980ac4e0 0x7f43980ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.878+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43a8016e50 con 0x7f43980ac4e0 2024-03-24T00:36:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.878+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43a80d5050 con 0x7f43b010e7a0 2024-03-24T00:36:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.879+0000 7f43b5200700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43980a29a0 0x7f43980a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f43a400b2e0 tx=0x7f43a401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.879+0000 7f43afe00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] conn(0x7f43980ac4e0 0x7f43980ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.879+0000 7f43afe00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] conn(0x7f43980ac4e0 0x7f43980ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.880+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== osd.2 v2:172.21.15.204:6800/823607592 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f43a8016e50 con 0x7f43980ac4e0 2024-03-24T00:36:05.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.898+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 --> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f43b0062d50 con 0x7f43980ac4e0 2024-03-24T00:36:05.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.898+0000 7f43ad600700 1 -- 172.21.15.123:0/1402880028 <== osd.2 v2:172.21.15.204:6800/823607592 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f43b0062d50 con 0x7f43980ac4e0 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] conn(0x7f43980ac4e0 msgr2=0x7f43980ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] conn(0x7f43980ac4e0 0x7f43980ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43980a29a0 msgr2=0x7f43980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43980a29a0 0x7f43980a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f43a400b2e0 tx=0x7f43a401a040 comp rx=0 tx=0).stop 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 msgr2=0x7f43b01a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 --2- 172.21.15.123:0/1402880028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43b010e7a0 0x7f43b01a3800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43a800ef30 tx=0x7f43a800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b5c00700 1 -- 172.21.15.123:0/1402880028 reap_dead start 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 shutdown_connections 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 >> 172.21.15.123:0/1402880028 conn(0x7f43b0075960 msgr2=0x7f43b0075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 shutdown_connections 2024-03-24T00:36:05.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:05.899+0000 7f43b7c00700 1 -- 172.21.15.123:0/1402880028 wait complete. 2024-03-24T00:36:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:06 smithi123 ceph-mon[31487]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:06 smithi177 ceph-mon[36034]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:06 smithi204 ceph-mon[36345]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:06.966 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:07.543 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:07.585 INFO:teuthology.orchestra.run.smithi123.stdout:201863462958 2024-03-24T00:36:07.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi123.stdout:352187318294 2024-03-24T00:36:08.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T00:36:08.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:08 smithi177 ceph-mon[36034]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:08.538 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:08 smithi123 ceph-mon[31487]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:08 smithi204 ceph-mon[36345]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:09.730 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:10 smithi177 ceph-mon[36034]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:10 smithi204 ceph-mon[36345]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:10 smithi123 ceph-mon[31487]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-24T00:36:11.640 INFO:teuthology.orchestra.run.smithi123.stdout:77309411398 2024-03-24T00:36:11.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T00:36:12.307 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:12 smithi123 ceph-mon[31487]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:12 smithi204 ceph-mon[36345]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:12 smithi177 ceph-mon[36034]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 -- 172.21.15.123:0/1322644340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 msgr2=0x7f103010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 --2- 172.21.15.123:0/1322644340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f103010f020 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f102800b0a0 tx=0x7f102800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 -- 172.21.15.123:0/1322644340 shutdown_connections 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 --2- 172.21.15.123:0/1322644340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f103010f020 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 --2- 172.21.15.123:0/1322644340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10301081e0 0x7f10301085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 --2- 172.21.15.123:0/1322644340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1030069e40 0x7f103006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 -- 172.21.15.123:0/1322644340 >> 172.21.15.123:0/1322644340 conn(0x7f1030077fc0 msgr2=0x7f10300783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 -- 172.21.15.123:0/1322644340 shutdown_connections 2024-03-24T00:36:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.457+0000 7f1035000700 1 -- 172.21.15.123:0/1322644340 wait complete. 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.458+0000 7f1035000700 1 Processor -- start 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 -- start start 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1030069e40 0x7f10301a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f10301a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10301081e0 0x7f103019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1030112480 con 0x7f10301081e0 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1030112300 con 0x7f103006a7e0 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f1035000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1030112600 con 0x7f1030069e40 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.459+0000 7f102d400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f10301a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f10301a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47674/0 (socket says 172.21.15.123:47674) 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 -- 172.21.15.123:0/303879978 learned_addr learned my addr 172.21.15.123:0/303879978 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102de00700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1030069e40 0x7f10301a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102e800700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10301081e0 0x7f103019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 -- 172.21.15.123:0/303879978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1030069e40 msgr2=0x7f10301a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1030069e40 0x7f10301a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 -- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10301081e0 msgr2=0x7f103019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10301081e0 0x7f103019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103019b070 con 0x7f103006a7e0 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f102d400700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f10301a0c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f101c00e9a0 tx=0x7f101c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.460+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c00ca10 con 0x7f103006a7e0 2024-03-24T00:36:14.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.461+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f101c00cbb0 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.461+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c018680 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.461+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103019b360 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.461+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10301a7620 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.461+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f103010dd50 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.464+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f101c010430 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f1026a00700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10140a29a0 0x7f10140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f101c0cf650 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f102de00700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10140a29a0 0x7f10140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f1026a00700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] conn(0x7f10140ac4e0 0x7f10140ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f101c01bec0 con 0x7f10140ac4e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.465+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f101c0cfa00 con 0x7f103006a7e0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.466+0000 7f102e800700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] conn(0x7f10140ac4e0 0x7f10140ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.466+0000 7f102de00700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10140a29a0 0x7f10140a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f103019ba50 tx=0x7f10200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.466+0000 7f102e800700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] conn(0x7f10140ac4e0 0x7f10140ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.467+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== osd.0 v2:172.21.15.123:6802/407676320 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f101c01bec0 con 0x7f10140ac4e0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.485+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 --> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1030103890 con 0x7f10140ac4e0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.485+0000 7f1026a00700 1 -- 172.21.15.123:0/303879978 <== osd.0 v2:172.21.15.123:6802/407676320 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1030103890 con 0x7f10140ac4e0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] conn(0x7f10140ac4e0 msgr2=0x7f10140ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] conn(0x7f10140ac4e0 0x7f10140ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10140a29a0 msgr2=0x7f10140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10140a29a0 0x7f10140a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f103019ba50 tx=0x7f10200073d0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 msgr2=0x7f10301a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 --2- 172.21.15.123:0/303879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103006a7e0 0x7f10301a0c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f101c00e9a0 tx=0x7f101c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f102e800700 1 -- 172.21.15.123:0/303879978 reap_dead start 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 shutdown_connections 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 >> 172.21.15.123:0/303879978 conn(0x7f1030077fc0 msgr2=0x7f103010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 shutdown_connections 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:13.486+0000 7f1035000700 1 -- 172.21.15.123:0/303879978 wait complete. 2024-03-24T00:36:14.546 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:14 smithi123 ceph-mon[31487]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:14 smithi204 ceph-mon[36345]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:14 smithi177 ceph-mon[36034]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:16.394 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:16 smithi123 ceph-mon[31487]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:16 smithi204 ceph-mon[36345]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:16 smithi177 ceph-mon[36034]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:17.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 -- 172.21.15.123:0/3928624072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 msgr2=0x7ff50c1049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 --2- 172.21.15.123:0/3928624072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1049c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff500009990 tx=0x7ff500009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 -- 172.21.15.123:0/3928624072 shutdown_connections 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 --2- 172.21.15.123:0/3928624072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff50c1096c0 0x7ff50c111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 --2- 172.21.15.123:0/3928624072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1049c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 --2- 172.21.15.123:0/3928624072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff50c104f90 0x7ff50c108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 -- 172.21.15.123:0/3928624072 >> 172.21.15.123:0/3928624072 conn(0x7ff50c0ffe70 msgr2=0x7ff50c102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.510+0000 7ff513a00700 1 -- 172.21.15.123:0/3928624072 shutdown_connections 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.511+0000 7ff513a00700 1 -- 172.21.15.123:0/3928624072 wait complete. 2024-03-24T00:36:17.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.512+0000 7ff513a00700 1 Processor -- start 2024-03-24T00:36:17.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.512+0000 7ff513a00700 1 -- start start 2024-03-24T00:36:17.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.512+0000 7ff513a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.512+0000 7ff513a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff50c104f90 0x7ff50c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff513a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff50c1096c0 0x7ff50c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff513a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c114710 con 0x7ff50c1096c0 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff513a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c114590 con 0x7ff50c1045e0 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff513a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c114890 con 0x7ff50c104f90 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59174/0 (socket says 172.21.15.123:59174) 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff50be00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff50c104f90 0x7ff50c1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 -- 172.21.15.123:0/4168300826 learned_addr learned my addr 172.21.15.123:0/4168300826 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff50c1096c0 0x7ff50c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 -- 172.21.15.123:0/4168300826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff50c104f90 msgr2=0x7ff50c1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff50c104f90 0x7ff50c1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 -- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff50c1096c0 msgr2=0x7ff50c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff50c1096c0 0x7ff50c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.513+0000 7ff511000700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c19b070 con 0x7ff50c1045e0 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff511000700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1a0750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff500005910 tx=0x7ff5000047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500027070 con 0x7ff50c1045e0 2024-03-24T00:36:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff50000fde0 con 0x7ff50c1045e0 2024-03-24T00:36:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5000173f0 con 0x7ff50c1045e0 2024-03-24T00:36:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c19b300 con 0x7ff50c1045e0 2024-03-24T00:36:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.514+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c1a7620 con 0x7ff50c1045e0 2024-03-24T00:36:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.515+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff50c062920 con 0x7ff50c1045e0 2024-03-24T00:36:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.517+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff50002a070 con 0x7ff50c1045e0 2024-03-24T00:36:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.518+0000 7ff509600700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4f40a29a0 0x7ff4f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff5000cfc40 con 0x7ff50c1045e0 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff50be00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4f40a29a0 0x7ff4f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff509600700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] conn(0x7ff4f40ac4e0 0x7ff4f40ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff50000fc30 con 0x7ff4f40ac4e0 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5000d0090 con 0x7ff50c1045e0 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff511a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] conn(0x7ff4f40ac4e0 0x7ff4f40ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.519+0000 7ff50be00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4f40a29a0 0x7ff4f40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff50c10ac30 tx=0x7ff4fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.520+0000 7ff511a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] conn(0x7ff4f40ac4e0 0x7ff4f40ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.521+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== osd.3 v2:172.21.15.123:6810/1321829693 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff50000fc30 con 0x7ff4f40ac4e0 2024-03-24T00:36:17.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.539+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 --> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff50c062d20 con 0x7ff4f40ac4e0 2024-03-24T00:36:17.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.539+0000 7ff509600700 1 -- 172.21.15.123:0/4168300826 <== osd.3 v2:172.21.15.123:6810/1321829693 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff50c062d20 con 0x7ff4f40ac4e0 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] conn(0x7ff4f40ac4e0 msgr2=0x7ff4f40ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] conn(0x7ff4f40ac4e0 0x7ff4f40ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4f40a29a0 msgr2=0x7ff4f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4f40a29a0 0x7ff4f40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff50c10ac30 tx=0x7ff4fc009260 comp rx=0 tx=0).stop 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 msgr2=0x7ff50c1a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 --2- 172.21.15.123:0/4168300826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff50c1045e0 0x7ff50c1a0750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff500005910 tx=0x7ff5000047b0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff511a00700 1 -- 172.21.15.123:0/4168300826 reap_dead start 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 shutdown_connections 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 >> 172.21.15.123:0/4168300826 conn(0x7ff50c0ffe70 msgr2=0x7ff50c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 shutdown_connections 2024-03-24T00:36:17.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.540+0000 7ff513a00700 1 -- 172.21.15.123:0/4168300826 wait complete. 2024-03-24T00:36:18.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 -- 172.21.15.123:0/3064274793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec10e7a0 msgr2=0x7f85ec10eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 --2- 172.21.15.123:0/3064274793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec10e7a0 0x7f85ec10eb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009990 tx=0x7f85dc009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 -- 172.21.15.123:0/3064274793 shutdown_connections 2024-03-24T00:36:18.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 --2- 172.21.15.123:0/3064274793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 0x7f85ec115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 --2- 172.21.15.123:0/3064274793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec10e7a0 0x7f85ec10eb80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 --2- 172.21.15.123:0/3064274793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85ec1087a0 0x7f85ec108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 -- 172.21.15.123:0/3064274793 >> 172.21.15.123:0/3064274793 conn(0x7f85ec075960 msgr2=0x7f85ec077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 -- 172.21.15.123:0/3064274793 shutdown_connections 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.040+0000 7f85f1200700 1 -- 172.21.15.123:0/3064274793 wait complete. 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.041+0000 7f85f1200700 1 Processor -- start 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 -- start start 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 0x7f85ec19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 0x7f85ec19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85ec10e7a0 0x7f85ec1a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec118b20 con 0x7f85ec109160 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1189a0 con 0x7f85ec1087a0 2024-03-24T00:36:18.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85f1200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec118ca0 con 0x7f85ec10e7a0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.042+0000 7f85e9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 0x7f85ec19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 0x7f85ec19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 0x7f85ec19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59186/0 (socket says 172.21.15.123:59186) 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 -- 172.21.15.123:0/4277932999 learned_addr learned my addr 172.21.15.123:0/4277932999 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 0x7f85ec19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49084/0 (socket says 172.21.15.123:49084) 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85ea800700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85ec10e7a0 0x7f85ec1a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 -- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85ec10e7a0 msgr2=0x7f85ec1a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85ec10e7a0 0x7f85ec1a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 -- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 msgr2=0x7f85ec19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85ec109160 0x7f85ec19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec1a3f40 con 0x7f85ec1087a0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e9e00700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 0x7f85ec19ef30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85dc000c00 tx=0x7f85dc00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.043+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc027070 con 0x7f85ec1087a0 2024-03-24T00:36:18.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.044+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85dc0056d0 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.044+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec1a41d0 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.044+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc022e10 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.044+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec1abab0 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.044+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85ec062920 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.046+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f85dc02a070 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e2a00700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f85d00a2920 0x7f85d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f85dc0cfc50 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e9400700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f85d00a2920 0x7f85d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e2a00700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] conn(0x7f85d00ac460 0x7f85d00ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85d00aefd0 con 0x7f85d00ac460 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.048+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85dc0d0000 con 0x7f85ec1087a0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.049+0000 7f85ea800700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] conn(0x7f85d00ac460 0x7f85d00ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.049+0000 7f85e9400700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f85d00a2920 0x7f85d00a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85ec10dd30 tx=0x7f85d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.049+0000 7f85ea800700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] conn(0x7f85d00ac460 0x7f85d00ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.050+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== osd.5 v2:172.21.15.204:6808/635738302 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f85d00aefd0 con 0x7f85d00ac460 2024-03-24T00:36:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.069+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 --> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85ec062da0 con 0x7f85d00ac460 2024-03-24T00:36:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.070+0000 7f85e2a00700 1 -- 172.21.15.123:0/4277932999 <== osd.5 v2:172.21.15.204:6808/635738302 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f85ec062da0 con 0x7f85d00ac460 2024-03-24T00:36:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.070+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] conn(0x7f85d00ac460 msgr2=0x7f85d00ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.070+0000 7f85f1200700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] conn(0x7f85d00ac460 0x7f85d00ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f85d00a2920 msgr2=0x7f85d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f85d00a2920 0x7f85d00a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85ec10dd30 tx=0x7f85d8009260 comp rx=0 tx=0).stop 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 msgr2=0x7f85ec19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 --2- 172.21.15.123:0/4277932999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ec1087a0 0x7f85ec19ef30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85dc000c00 tx=0x7f85dc00bab0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85ea800700 1 -- 172.21.15.123:0/4277932999 reap_dead start 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 shutdown_connections 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 >> 172.21.15.123:0/4277932999 conn(0x7f85ec075960 msgr2=0x7f85ec075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 shutdown_connections 2024-03-24T00:36:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.071+0000 7f85f1200700 1 -- 172.21.15.123:0/4277932999 wait complete. 2024-03-24T00:36:18.341 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:18 smithi123 ceph-mon[31487]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:18 smithi204 ceph-mon[36345]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:18 smithi177 ceph-mon[36034]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:18.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.808+0000 7f1608c00700 1 -- 172.21.15.123:0/4163015345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 msgr2=0x7f160410ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 --2- 172.21.15.123:0/4163015345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160410ef70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00b0a0 tx=0x7f15fc00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 -- 172.21.15.123:0/4163015345 shutdown_connections 2024-03-24T00:36:18.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 --2- 172.21.15.123:0/4163015345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160410ef70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 --2- 172.21.15.123:0/4163015345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16041022c0 0x7f16041026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 --2- 172.21.15.123:0/4163015345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1604102c70 0x7f1604106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 -- 172.21.15.123:0/4163015345 >> 172.21.15.123:0/4163015345 conn(0x7f16040fdbb0 msgr2=0x7f16040fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 -- 172.21.15.123:0/4163015345 shutdown_connections 2024-03-24T00:36:18.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.809+0000 7f1608c00700 1 -- 172.21.15.123:0/4163015345 wait complete. 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 Processor -- start 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 -- start start 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16041022c0 0x7f16041a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1604102c70 0x7f16041a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604112390 con 0x7f1604102c70 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604112210 con 0x7f16041073a0 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.810+0000 7f1608c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604112510 con 0x7f16041022c0 2024-03-24T00:36:18.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1602a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1604102c70 0x7f16041a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1602a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1604102c70 0x7f16041a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49104/0 (socket says 172.21.15.123:49104) 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1602a00700 1 -- 172.21.15.123:0/3854253068 learned_addr learned my addr 172.21.15.123:0/3854253068 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603400700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16041022c0 0x7f16041a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 -- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16041022c0 msgr2=0x7f16041a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16041022c0 0x7f16041a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 -- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1604102c70 msgr2=0x7f16041a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1604102c70 0x7f16041a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.811+0000 7f1603e00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160419aff0 con 0x7f16041073a0 2024-03-24T00:36:18.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f1603e00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160419a890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15fc009bc0 tx=0x7f15fc004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc021070 con 0x7f16041073a0 2024-03-24T00:36:18.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15fc007b80 con 0x7f16041073a0 2024-03-24T00:36:18.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc0052a0 con 0x7f16041073a0 2024-03-24T00:36:18.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160419b280 con 0x7f16041073a0 2024-03-24T00:36:18.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.812+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16041a75a0 con 0x7f16041073a0 2024-03-24T00:36:18.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.813+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16040fef00 con 0x7f16041073a0 2024-03-24T00:36:18.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.816+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f15fc024070 con 0x7f16041073a0 2024-03-24T00:36:18.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.817+0000 7f15fbe00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f15e80a29a0 0x7f15e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f15fc00f020 con 0x7f16041073a0 2024-03-24T00:36:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f1603400700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f15e80a29a0 0x7f15e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f15fbe00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] conn(0x7f15e80ac4e0 0x7f15e80ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15fc020ec0 con 0x7f15e80ac4e0 2024-03-24T00:36:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15fc0d2370 con 0x7f16041073a0 2024-03-24T00:36:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.818+0000 7f1602a00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] conn(0x7f15e80ac4e0 0x7f15e80ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.819+0000 7f1603400700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f15e80a29a0 0x7f15e80a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f160419b9d0 tx=0x7f15f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.819+0000 7f1602a00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] conn(0x7f15e80ac4e0 0x7f15e80ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.820+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== osd.8 v2:172.21.15.204:6816/4011283541 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15fc020ec0 con 0x7f15e80ac4e0 2024-03-24T00:36:18.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.838+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 --> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16041037a0 con 0x7f15e80ac4e0 2024-03-24T00:36:18.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.838+0000 7f15fbe00700 1 -- 172.21.15.123:0/3854253068 <== osd.8 v2:172.21.15.204:6816/4011283541 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f16041037a0 con 0x7f15e80ac4e0 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] conn(0x7f15e80ac4e0 msgr2=0x7f15e80ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] conn(0x7f15e80ac4e0 0x7f15e80ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f15e80a29a0 msgr2=0x7f15e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f15e80a29a0 0x7f15e80a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f160419b9d0 tx=0x7f15f4009260 comp rx=0 tx=0).stop 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 msgr2=0x7f160419a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 --2- 172.21.15.123:0/3854253068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041073a0 0x7f160419a890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15fc009bc0 tx=0x7f15fc004d00 comp rx=0 tx=0).stop 2024-03-24T00:36:18.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1603e00700 1 -- 172.21.15.123:0/3854253068 reap_dead start 2024-03-24T00:36:18.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 shutdown_connections 2024-03-24T00:36:18.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 >> 172.21.15.123:0/3854253068 conn(0x7f16040fdbb0 msgr2=0x7f16040fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 shutdown_connections 2024-03-24T00:36:18.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.839+0000 7f1608c00700 1 -- 172.21.15.123:0/3854253068 wait complete. 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 -- 172.21.15.123:0/3662446514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f0074ce0 msgr2=0x7f16f010e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 --2- 172.21.15.123:0/3662446514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f0074ce0 0x7f16f010e650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009990 tx=0x7f16dc009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 -- 172.21.15.123:0/3662446514 shutdown_connections 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 --2- 172.21.15.123:0/3662446514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 0x7f16f0110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 --2- 172.21.15.123:0/3662446514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16f00743c0 0x7f16f00747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 --2- 172.21.15.123:0/3662446514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f0074ce0 0x7f16f010e650 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 -- 172.21.15.123:0/3662446514 >> 172.21.15.123:0/3662446514 conn(0x7f16f00792f0 msgr2=0x7f16f0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 -- 172.21.15.123:0/3662446514 shutdown_connections 2024-03-24T00:36:18.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.898+0000 7f16f5600700 1 -- 172.21.15.123:0/3662446514 wait complete. 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.899+0000 7f16f5600700 1 Processor -- start 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.899+0000 7f16f5600700 1 -- start start 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 0x7f16f01a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16f0074ce0 0x7f16f01a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 0x7f16f019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f01147a0 con 0x7f16f010eb90 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0114620 con 0x7f16f00743c0 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16f5600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0114920 con 0x7f16f0074ce0 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 0x7f16f01a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16ef400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 0x7f16f019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 0x7f16f01a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59218/0 (socket says 172.21.15.123:59218) 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 -- 172.21.15.123:0/4100263009 learned_addr learned my addr 172.21.15.123:0/4100263009 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16ee000700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16f0074ce0 0x7f16f01a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 -- 172.21.15.123:0/4100263009 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16f0074ce0 msgr2=0x7f16f01a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16f0074ce0 0x7f16f01a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 -- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 msgr2=0x7f16f019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 0x7f16f019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f019b1c0 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16ef400700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f010eb90 0x7f16f019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.900+0000 7f16eea00700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 0x7f16f01a0850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16e000e9c0 tx=0x7f16e000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.901+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000cbd0 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.901+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e000cd70 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.901+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f019b4b0 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.901+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0010430 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.901+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f01a7720 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.902+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16f00ffd20 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.903+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f16e0004750 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16e7400700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f16d40a29a0 0x7f16d40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f16e00cee80 con 0x7f16f00743c0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16ee000700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f16d40a29a0 0x7f16d40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16e7400700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] conn(0x7f16d40ac4e0 0x7f16d40ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16d40af050 con 0x7f16d40ac4e0 2024-03-24T00:36:18.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e00d14a0 con 0x7f16f00743c0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.905+0000 7f16ef400700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] conn(0x7f16d40ac4e0 0x7f16d40ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.906+0000 7f16ee000700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f16d40a29a0 0x7f16d40a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16f0101c90 tx=0x7f16dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.906+0000 7f16ef400700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] conn(0x7f16d40ac4e0 0x7f16d40ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.907+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== osd.10 v2:172.21.15.177:6824/2089800664 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f16d40af050 con 0x7f16d40ac4e0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.925+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 --> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16f0075af0 con 0x7f16d40ac4e0 2024-03-24T00:36:18.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.925+0000 7f16e7400700 1 -- 172.21.15.123:0/4100263009 <== osd.10 v2:172.21.15.177:6824/2089800664 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f16f0075af0 con 0x7f16d40ac4e0 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] conn(0x7f16d40ac4e0 msgr2=0x7f16d40ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] conn(0x7f16d40ac4e0 0x7f16d40ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f16d40a29a0 msgr2=0x7f16d40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f16d40a29a0 0x7f16d40a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16f0101c90 tx=0x7f16dc005fd0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 msgr2=0x7f16f01a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 --2- 172.21.15.123:0/4100263009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16f00743c0 0x7f16f01a0850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16e000e9c0 tx=0x7f16e000ecd0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16ef400700 1 -- 172.21.15.123:0/4100263009 reap_dead start 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 shutdown_connections 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 >> 172.21.15.123:0/4100263009 conn(0x7f16f00792f0 msgr2=0x7f16f00ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 shutdown_connections 2024-03-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:18.926+0000 7f16f5600700 1 -- 172.21.15.123:0/4100263009 wait complete. 2024-03-24T00:36:19.824 INFO:teuthology.orchestra.run.smithi123.stdout:317827579932 2024-03-24T00:36:19.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T00:36:20.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.667+0000 7f8941400700 1 -- 172.21.15.123:0/3275740550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 msgr2=0x7f893c1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.667+0000 7f8941400700 1 --2- 172.21.15.123:0/3275740550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 0x7f893c1048a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f892c009920 tx=0x7f892c009c30 comp rx=0 tx=0).stop 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 -- 172.21.15.123:0/3275740550 shutdown_connections 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 --2- 172.21.15.123:0/3275740550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c1095a0 0x7f893c111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 --2- 172.21.15.123:0/3275740550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 0x7f893c1048a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 --2- 172.21.15.123:0/3275740550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 -- 172.21.15.123:0/3275740550 >> 172.21.15.123:0/3275740550 conn(0x7f893c0ffd70 msgr2=0x7f893c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 -- 172.21.15.123:0/3275740550 shutdown_connections 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.668+0000 7f8941400700 1 -- 172.21.15.123:0/3275740550 wait complete. 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.670+0000 7f8941400700 1 Processor -- start 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.670+0000 7f8941400700 1 -- start start 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.670+0000 7f8941400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 0x7f893c1a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.670+0000 7f8941400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.670+0000 7f8941400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c1095a0 0x7f893c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f8941400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c114580 con 0x7f893c1095a0 2024-03-24T00:36:20.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f8941400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c114400 con 0x7f893c104e70 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f8941400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c114700 con 0x7f893c1044c0 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c1a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59236/0 (socket says 172.21.15.123:59236) 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 -- 172.21.15.123:0/2430115788 learned_addr learned my addr 172.21.15.123:0/2430115788 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893be00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 0x7f893c1a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f8940a00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c1095a0 0x7f893c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 -- 172.21.15.123:0/2430115788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 msgr2=0x7f893c1a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f893c1044c0 0x7f893c1a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 -- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c1095a0 msgr2=0x7f893c19a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c1095a0 0x7f893c19a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.671+0000 7f893b400700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c19b060 con 0x7f893c104e70 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f893b400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c1a0c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f892800ec70 tx=0x7f892800ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892800cc90 con 0x7f893c104e70 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f892800ce30 con 0x7f893c104e70 2024-03-24T00:36:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c19b350 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928010470 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.672+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c1a7610 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.673+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f893c062920 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.675+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8928004000 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f8938c00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89200a29a0 0x7f89200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89280cee70 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f893be00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89200a29a0 0x7f89200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f8938c00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] conn(0x7f89200ac4e0 0x7f89200ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8928020e50 con 0x7f89200ac4e0 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.676+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89280d1530 con 0x7f893c104e70 2024-03-24T00:36:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.677+0000 7f8940a00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] conn(0x7f89200ac4e0 0x7f89200ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.677+0000 7f893be00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89200a29a0 0x7f89200a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f892c0098f0 tx=0x7f892c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.677+0000 7f8940a00700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] conn(0x7f89200ac4e0 0x7f89200ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.678+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== osd.4 v2:172.21.15.177:6808/2757515144 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8928020e50 con 0x7f89200ac4e0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.696+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 --> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f893c062d20 con 0x7f89200ac4e0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8938c00700 1 -- 172.21.15.123:0/2430115788 <== osd.4 v2:172.21.15.177:6808/2757515144 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f893c062d20 con 0x7f89200ac4e0 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] conn(0x7f89200ac4e0 msgr2=0x7f89200ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] conn(0x7f89200ac4e0 0x7f89200ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89200a29a0 msgr2=0x7f89200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89200a29a0 0x7f89200a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f892c0098f0 tx=0x7f892c019040 comp rx=0 tx=0).stop 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 msgr2=0x7f893c1a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.697+0000 7f8941400700 1 --2- 172.21.15.123:0/2430115788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c104e70 0x7f893c1a0c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f892800ec70 tx=0x7f892800ef80 comp rx=0 tx=0).stop 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.698+0000 7f8940a00700 1 -- 172.21.15.123:0/2430115788 reap_dead start 2024-03-24T00:36:20.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.698+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 shutdown_connections 2024-03-24T00:36:20.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.698+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 >> 172.21.15.123:0/2430115788 conn(0x7f893c0ffd70 msgr2=0x7f893c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.698+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 shutdown_connections 2024-03-24T00:36:20.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:19.698+0000 7f8941400700 1 -- 172.21.15.123:0/2430115788 wait complete. 2024-03-24T00:36:20.450 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:20 smithi123 ceph-mon[31487]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:20 smithi204 ceph-mon[36345]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:20 smithi177 ceph-mon[36034]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:111669149764 2024-03-24T00:36:20.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T00:36:20.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 -- 172.21.15.123:0/117585928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 msgr2=0x7fb6541112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 --2- 172.21.15.123:0/117585928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb6541112b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b0a0 tx=0x7fb64c00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 -- 172.21.15.123:0/117585928 shutdown_connections 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 --2- 172.21.15.123:0/117585928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb6541112b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 --2- 172.21.15.123:0/117585928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb654104600 0x7fb6541049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 --2- 172.21.15.123:0/117585928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 0x7fb654109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 -- 172.21.15.123:0/117585928 >> 172.21.15.123:0/117585928 conn(0x7fb6540ffe90 msgr2=0x7fb6541022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 -- 172.21.15.123:0/117585928 shutdown_connections 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.197+0000 7fb65b800700 1 -- 172.21.15.123:0/117585928 wait complete. 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.198+0000 7fb65b800700 1 Processor -- start 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.198+0000 7fb65b800700 1 -- start start 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.198+0000 7fb65b800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb654104600 0x7fb6541a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.198+0000 7fb65b800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 0x7fb6541a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb65b800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb65419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb65b800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb654114720 con 0x7fb654104fb0 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb65b800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541145a0 con 0x7fb6541096e0 2024-03-24T00:36:20.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb65b800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541148a0 con 0x7fb654104600 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb653e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 0x7fb6541a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb65419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb653e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 0x7fb6541a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49044/0 (socket says 172.21.15.123:49044) 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb653e00700 1 -- 172.21.15.123:0/979228409 learned_addr learned my addr 172.21.15.123:0/979228409 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb658e00700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb654104600 0x7fb6541a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 -- 172.21.15.123:0/979228409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb654104600 msgr2=0x7fb6541a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb654104600 0x7fb6541a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 -- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 msgr2=0x7fb6541a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb654104fb0 0x7fb6541a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65419b120 con 0x7fb6541096e0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.199+0000 7fb659800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb65419a9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0062a0 tx=0x7fb64c004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.200+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c021070 con 0x7fb6541096e0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.200+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb64c003c00 con 0x7fb6541096e0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.200+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65419b3b0 con 0x7fb6541096e0 2024-03-24T00:36:20.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.200+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c004750 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.200+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6541a7680 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.201+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6540628a0 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.203+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb64c003db0 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb651600700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb63c0a29a0 0x7fb63c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb64c0cfd60 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb658e00700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb63c0a29a0 0x7fb63c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb651600700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] conn(0x7fb63c0ac4e0 0x7fb63c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64c02ee50 con 0x7fb63c0ac4e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.204+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb64c0d0110 con 0x7fb6541096e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.205+0000 7fb653e00700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] conn(0x7fb63c0ac4e0 0x7fb63c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.205+0000 7fb658e00700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb63c0a29a0 0x7fb63c0a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb654109410 tx=0x7fb64800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.205+0000 7fb653e00700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] conn(0x7fb63c0ac4e0 0x7fb63c0ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.206+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== osd.9 v2:172.21.15.123:6826/3491687003 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb64c02ee50 con 0x7fb63c0ac4e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.225+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 --> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb654062d20 con 0x7fb63c0ac4e0 2024-03-24T00:36:20.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.225+0000 7fb651600700 1 -- 172.21.15.123:0/979228409 <== osd.9 v2:172.21.15.123:6826/3491687003 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb654062d20 con 0x7fb63c0ac4e0 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] conn(0x7fb63c0ac4e0 msgr2=0x7fb63c0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] conn(0x7fb63c0ac4e0 0x7fb63c0ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb63c0a29a0 msgr2=0x7fb63c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb63c0a29a0 0x7fb63c0a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb654109410 tx=0x7fb64800b410 comp rx=0 tx=0).stop 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 msgr2=0x7fb65419a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 --2- 172.21.15.123:0/979228409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6541096e0 0x7fb65419a9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0062a0 tx=0x7fb64c004cf0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb659800700 1 -- 172.21.15.123:0/979228409 reap_dead start 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 shutdown_connections 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 >> 172.21.15.123:0/979228409 conn(0x7fb6540ffe90 msgr2=0x7fb654100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 shutdown_connections 2024-03-24T00:36:20.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:17.226+0000 7fb65b800700 1 -- 172.21.15.123:0/979228409 wait complete. 2024-03-24T00:36:21.520 INFO:teuthology.orchestra.run.smithi123.stdout:146028888126 2024-03-24T00:36:21.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T00:36:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:22 smithi204 ceph-mon[36345]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:22.605 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:22 smithi123 ceph-mon[31487]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:22.663 INFO:teuthology.orchestra.run.smithi123.stdout:292057776162 2024-03-24T00:36:22.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T00:36:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:22 smithi177 ceph-mon[36034]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:23.055 INFO:teuthology.orchestra.run.smithi123.stdout:34359738450 2024-03-24T00:36:23.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T00:36:23.593 INFO:teuthology.orchestra.run.smithi123.stdout:257698037799 2024-03-24T00:36:23.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T00:36:23.797 INFO:teuthology.orchestra.run.smithi123.stdout:167503724600 2024-03-24T00:36:23.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.395+0000 7f5d95400700 1 -- 172.21.15.123:0/708462306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90107510 msgr2=0x7f5d9010f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.395+0000 7f5d95400700 1 --2- 172.21.15.123:0/708462306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90107510 0x7f5d9010f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d8800b0a0 tx=0x7f5d8800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 -- 172.21.15.123:0/708462306 shutdown_connections 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 --2- 172.21.15.123:0/708462306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90107510 0x7f5d9010f0e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 --2- 172.21.15.123:0/708462306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d90102440 0x7f5d90102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 --2- 172.21.15.123:0/708462306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90102df0 0x7f5d90106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 -- 172.21.15.123:0/708462306 >> 172.21.15.123:0/708462306 conn(0x7f5d900792f0 msgr2=0x7f5d90079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 -- 172.21.15.123:0/708462306 shutdown_connections 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.396+0000 7f5d95400700 1 -- 172.21.15.123:0/708462306 wait complete. 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.397+0000 7f5d95400700 1 Processor -- start 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.397+0000 7f5d95400700 1 -- start start 2024-03-24T00:36:23.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d90102440 0x7f5d9019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 0x7f5d9019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90107510 0x7f5d90198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90112580 con 0x7f5d90107510 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90112400 con 0x7f5d90102df0 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d95400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90112700 con 0x7f5d90102440 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d87e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 0x7f5d9019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d87e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 0x7f5d9019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59258/0 (socket says 172.21.15.123:59258) 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d87e00700 1 -- 172.21.15.123:0/976104965 learned_addr learned my addr 172.21.15.123:0/976104965 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d8e800700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d90102440 0x7f5d9019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.398+0000 7f5d8f200700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90107510 0x7f5d90198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 -- 172.21.15.123:0/976104965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d90102440 msgr2=0x7f5d9019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d90102440 0x7f5d9019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 -- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90107510 msgr2=0x7f5d90198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90107510 0x7f5d90198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90199020 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87e00700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 0x7f5d9019ebb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e970 tx=0x7f5d7800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78004a30 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d7800ccc0 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78005650 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d90199310 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.399+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d901a54d0 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.400+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d90062860 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.402+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5d78004bd0 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.403+0000 7f5d87400700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5d740a29a0 0x7f5d740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.403+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5d780ced70 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.403+0000 7f5d8e800700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5d740a29a0 0x7f5d740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d87400700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] conn(0x7f5d740ac4e0 0x7f5d740ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d78021e50 con 0x7f5d740ac4e0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d780d1390 con 0x7f5d90102df0 2024-03-24T00:36:23.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d8f200700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] conn(0x7f5d740ac4e0 0x7f5d740ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d8e800700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5d740a29a0 0x7f5d740a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009c60 tx=0x7f5d80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.404+0000 7f5d8f200700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] conn(0x7f5d740ac4e0 0x7f5d740ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.406+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== osd.1 v2:172.21.15.177:6800/1151434087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5d78021e50 con 0x7f5d740ac4e0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.423+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 --> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d90062d10 con 0x7f5d740ac4e0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d87400700 1 -- 172.21.15.123:0/976104965 <== osd.1 v2:172.21.15.177:6800/1151434087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5d90062d10 con 0x7f5d740ac4e0 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] conn(0x7f5d740ac4e0 msgr2=0x7f5d740ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] conn(0x7f5d740ac4e0 0x7f5d740ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:23.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5d740a29a0 msgr2=0x7f5d740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5d740a29a0 0x7f5d740a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009c60 tx=0x7f5d80009260 comp rx=0 tx=0).stop 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 msgr2=0x7f5d9019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.424+0000 7f5d95400700 1 --2- 172.21.15.123:0/976104965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d90102df0 0x7f5d9019ebb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e970 tx=0x7f5d7800ec80 comp rx=0 tx=0).stop 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.425+0000 7f5d8f200700 1 -- 172.21.15.123:0/976104965 reap_dead start 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.425+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 shutdown_connections 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.425+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 >> 172.21.15.123:0/976104965 conn(0x7f5d900792f0 msgr2=0x7f5d90100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.425+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 shutdown_connections 2024-03-24T00:36:23.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:23.425+0000 7f5d95400700 1 -- 172.21.15.123:0/976104965 wait complete. 2024-03-24T00:36:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:24 smithi204 ceph-mon[36345]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:24 smithi123 ceph-mon[31487]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:24 smithi177 ceph-mon[36034]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:24.795 INFO:teuthology.orchestra.run.smithi123.stdout:55834574927 2024-03-24T00:36:24.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 -- 172.21.15.123:0/1259633608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2000743c0 msgr2=0x7fb2000747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 --2- 172.21.15.123:0/1259633608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2000743c0 0x7fb2000747a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009990 tx=0x7fb1f4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 -- 172.21.15.123:0/1259633608 shutdown_connections 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 --2- 172.21.15.123:0/1259633608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb200113360 0x7fb200115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 --2- 172.21.15.123:0/1259633608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2000743c0 0x7fb2000747a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 --2- 172.21.15.123:0/1259633608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb200074ce0 0x7fb200112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.245+0000 7fb208000700 1 -- 172.21.15.123:0/1259633608 >> 172.21.15.123:0/1259633608 conn(0x7fb2000792f0 msgr2=0x7fb200079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.246+0000 7fb208000700 1 -- 172.21.15.123:0/1259633608 shutdown_connections 2024-03-24T00:36:25.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.247+0000 7fb208000700 1 -- 172.21.15.123:0/1259633608 wait complete. 2024-03-24T00:36:25.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.248+0000 7fb208000700 1 Processor -- start 2024-03-24T00:36:25.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.248+0000 7fb208000700 1 -- start start 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000743c0 0x7fb20019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb200074ce0 0x7fb20019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb200113360 0x7fb2001a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200118b90 con 0x7fb2000743c0 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200118a10 con 0x7fb200074ce0 2024-03-24T00:36:25.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb208000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200118d10 con 0x7fb200113360 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.249+0000 7fb204c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb200074ce0 0x7fb20019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb204c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb200074ce0 0x7fb20019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59272/0 (socket says 172.21.15.123:59272) 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb204c00700 1 -- 172.21.15.123:0/1475107057 learned_addr learned my addr 172.21.15.123:0/1475107057 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb205600700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000743c0 0x7fb20019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb206000700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb200113360 0x7fb2001a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb205600700 1 -- 172.21.15.123:0/1475107057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb200113360 msgr2=0x7fb2001a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb205600700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb200113360 0x7fb2001a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.250+0000 7fb205600700 1 -- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb200074ce0 msgr2=0x7fb20019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.251+0000 7fb205600700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb200074ce0 0x7fb20019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.251+0000 7fb205600700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2001a3fa0 con 0x7fb2000743c0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.252+0000 7fb205600700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000743c0 0x7fb20019efe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.252+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4026070 con 0x7fb2000743c0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.252+0000 7fb208000700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2001a4230 con 0x7fb2000743c0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.253+0000 7fb208000700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2001abb10 con 0x7fb2000743c0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.253+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1f4021410 con 0x7fb2000743c0 2024-03-24T00:36:25.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.253+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4017650 con 0x7fb2000743c0 2024-03-24T00:36:25.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.254+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb1dc001030 con 0x7fb2000743c0 2024-03-24T00:36:25.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.258+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb1f4029070 con 0x7fb2000743c0 2024-03-24T00:36:25.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb1fe000700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb1e80a2920 0x7fb1e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:25.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb1f40cfd00 con 0x7fb2000743c0 2024-03-24T00:36:25.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb204c00700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb1e80a2920 0x7fb1e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:25.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb1fe000700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] conn(0x7fb1e80ac460 0x7fb1e80ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1f4023ec0 con 0x7fb1e80ac460 2024-03-24T00:36:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.260+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1f40d0150 con 0x7fb2000743c0 2024-03-24T00:36:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.261+0000 7fb206000700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] conn(0x7fb1e80ac460 0x7fb1e80ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.261+0000 7fb206000700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] conn(0x7fb1e80ac460 0x7fb1e80ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.261+0000 7fb204c00700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb1e80a2920 0x7fb1e80a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb2001151d0 tx=0x7fb1ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:25.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.262+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== osd.7 v2:172.21.15.177:6816/2491926780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb1f4023ec0 con 0x7fb1e80ac460 2024-03-24T00:36:25.266 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:25.269 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:25.287 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:25.306 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:25.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.305+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 --> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb1dc002e60 con 0x7fb1e80ac460 2024-03-24T00:36:25.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.307+0000 7fb1fe000700 1 -- 172.21.15.123:0/1475107057 <== osd.7 v2:172.21.15.177:6816/2491926780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb1dc002e60 con 0x7fb1e80ac460 2024-03-24T00:36:25.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.307+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] conn(0x7fb1e80ac460 msgr2=0x7fb1e80ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.307+0000 7fb1f3400700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] conn(0x7fb1e80ac460 0x7fb1e80ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.307+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb1e80a2920 msgr2=0x7fb1e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.307+0000 7fb1f3400700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb1e80a2920 0x7fb1e80a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb2001151d0 tx=0x7fb1ec009260 comp rx=0 tx=0).stop 2024-03-24T00:36:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.308+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000743c0 msgr2=0x7fb20019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.308+0000 7fb1f3400700 1 --2- 172.21.15.123:0/1475107057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000743c0 0x7fb20019efe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f40047b0 comp rx=0 tx=0).stop 2024-03-24T00:36:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.308+0000 7fb206000700 1 -- 172.21.15.123:0/1475107057 reap_dead start 2024-03-24T00:36:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:25.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.308+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 shutdown_connections 2024-03-24T00:36:25.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.308+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 >> 172.21.15.123:0/1475107057 conn(0x7fb2000792f0 msgr2=0x7fb2001120b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:25.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.309+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 shutdown_connections 2024-03-24T00:36:25.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:25.309+0000 7fb1f3400700 1 -- 172.21.15.123:0/1475107057 wait complete. 2024-03-24T00:36:25.312 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:26.510 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:26 smithi123 ceph-mon[31487]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:26 smithi204 ceph-mon[36345]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:26 smithi177 ceph-mon[36034]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:28.198 INFO:teuthology.orchestra.run.smithi123.stdout:223338299439 2024-03-24T00:36:28.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T00:36:28.407 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:28 smithi123 ceph-mon[31487]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:28 smithi204 ceph-mon[36345]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:28 smithi177 ceph-mon[36034]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:29.960 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:29.960 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:29.963 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:30.564 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:30 smithi123 ceph-mon[31487]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:30 smithi204 ceph-mon[36345]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:30 smithi177 ceph-mon[36034]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:32.331 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:32 smithi123 ceph-mon[31487]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-24T00:36:32.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:32 smithi177 ceph-mon[36034]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-24T00:36:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:32 smithi204 ceph-mon[36345]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-24T00:36:34.326 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:34 smithi123 ceph-mon[31487]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:34.326 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:34 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4027636768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:36:34.326 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:34 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:34 smithi177 ceph-mon[36034]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:34 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4027636768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:36:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:34 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:34 smithi204 ceph-mon[36345]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:34 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4027636768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:36:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:34 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:35.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 -- 172.21.15.123:0/1054436779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df4107080 msgr2=0x7f7df4069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 --2- 172.21.15.123:0/1054436779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df4107080 0x7f7df4069e40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009990 tx=0x7f7de4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 -- 172.21.15.123:0/1054436779 shutdown_connections 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 --2- 172.21.15.123:0/1054436779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df4105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 --2- 172.21.15.123:0/1054436779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df4107080 0x7f7df4069e40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 --2- 172.21.15.123:0/1054436779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df406a380 0x7f7df406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 -- 172.21.15.123:0/1054436779 >> 172.21.15.123:0/1054436779 conn(0x7f7df4077fc0 msgr2=0x7f7df40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.443+0000 7f7df9200700 1 -- 172.21.15.123:0/1054436779 shutdown_connections 2024-03-24T00:36:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.444+0000 7f7df9200700 1 -- 172.21.15.123:0/1054436779 wait complete. 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 Processor -- start 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 -- start start 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 0x7f7df419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 0x7f7df41a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4118a80 con 0x7f7df4104b90 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4118900 con 0x7f7df4107080 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df9200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4118c00 con 0x7f7df406a380 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df1e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 0x7f7df419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.445+0000 7f7df2800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 0x7f7df41a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45604/0 (socket says 172.21.15.123:45604) 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df1e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 0x7f7df419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:55422/0 (socket says 172.21.15.123:55422) 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df1400700 1 -- 172.21.15.123:0/3381287789 learned_addr learned my addr 172.21.15.123:0/3381287789 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 -- 172.21.15.123:0/3381287789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 msgr2=0x7f7df419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 0x7f7df419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 -- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 msgr2=0x7f7df419f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 -- 172.21.15.123:0/3381287789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df41a4000 con 0x7f7df4107080 2024-03-24T00:36:35.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df1400700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.446+0000 7f7df2800700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 0x7f7df41a38c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00efb0 tx=0x7f7dec00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.447+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec003a00 con 0x7f7df4107080 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.447+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dec003ba0 con 0x7f7df4107080 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.447+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec004850 con 0x7f7df4107080 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.447+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df41a42f0 con 0x7f7df4107080 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.447+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df41abb70 con 0x7f7df4107080 2024-03-24T00:36:35.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.448+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df4068e50 con 0x7f7df4107080 2024-03-24T00:36:35.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.450+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7dec012050 con 0x7f7df4107080 2024-03-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.452+0000 7f7deaa00700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 0x7f7dd80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.452+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7dec0ce3e0 con 0x7f7df4107080 2024-03-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.452+0000 7f7df1e00700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 0x7f7dd80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.453+0000 7f7df1e00700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 0x7f7dd80a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7de400b2e0 tx=0x7f7de4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.456+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7dec022060 con 0x7f7df4107080 2024-03-24T00:36:35.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 -- 172.21.15.123:0/4265102318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 msgr2=0x7f7a2010e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 --2- 172.21.15.123:0/4265102318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a2010e510 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009990 tx=0x7f7a08009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 -- 172.21.15.123:0/4265102318 shutdown_connections 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 --2- 172.21.15.123:0/4265102318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a2010ea50 0x7f7a20110e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 --2- 172.21.15.123:0/4265102318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a20074560 0x7f7a20074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 --2- 172.21.15.123:0/4265102318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a2010e510 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 -- 172.21.15.123:0/4265102318 >> 172.21.15.123:0/4265102318 conn(0x7f7a200fdc60 msgr2=0x7f7a20100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 -- 172.21.15.123:0/4265102318 shutdown_connections 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.335+0000 7f7a26000700 1 -- 172.21.15.123:0/4265102318 wait complete. 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.336+0000 7f7a26000700 1 Processor -- start 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.336+0000 7f7a26000700 1 -- start start 2024-03-24T00:36:35.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 0x7f7a201a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a201a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a2010ea50 0x7f7a2019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20114740 con 0x7f7a20074560 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a201145c0 con 0x7f7a20074e80 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a26000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a201148c0 con 0x7f7a2010ea50 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a16a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a201a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a16a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a201a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:36720/0 (socket says 172.21.15.123:36720) 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a16a00700 1 -- 172.21.15.123:0/4027636768 learned_addr learned my addr 172.21.15.123:0/4027636768 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.337+0000 7f7a1f400700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 0x7f7a201a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 -- 172.21.15.123:0/4027636768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a2010ea50 msgr2=0x7f7a2019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a2010ea50 0x7f7a2019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 -- 172.21.15.123:0/4027636768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 msgr2=0x7f7a201a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a201a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 -- 172.21.15.123:0/4027636768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2019b120 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.338+0000 7f7a1f400700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 0x7f7a201a0800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000d950 tx=0x7f7a1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.339+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10017070 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.339+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2019b410 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.339+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a201a76d0 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.339+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a10009e10 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.339+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10010e20 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.341+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79fc005330 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.344+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7a1001a070 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.345+0000 7f7a1cc00700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 0x7f7a040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.345+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7a100ce410 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.346+0000 7f7a16a00700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 0x7f7a040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.346+0000 7f7a16a00700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 0x7f7a040a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a20101800 tx=0x7f7a0800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.350+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a1009b300 con 0x7f7a20074560 2024-03-24T00:36:35.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.589+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f79fc0058f0 con 0x7f7a20074560 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.590+0000 7f7a1cc00700 1 -- 172.21.15.123:0/4027636768 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a1009ac00 con 0x7f7a20074560 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stdout:77309411403 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 msgr2=0x7f7a040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 0x7f7a040a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a20101800 tx=0x7f7a0800b260 comp rx=0 tx=0).stop 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 msgr2=0x7f7a201a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 0x7f7a201a0800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000d950 tx=0x7f7a1000dc60 comp rx=0 tx=0).stop 2024-03-24T00:36:35.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 shutdown_connections 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a040a29a0 0x7f7a040a4e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a2010ea50 0x7f7a2019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a20074560 0x7f7a201a0800 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 --2- 172.21.15.123:0/4027636768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a20074e80 0x7f7a201a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 >> 172.21.15.123:0/4027636768 conn(0x7f7a200fdc60 msgr2=0x7f7a200fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 shutdown_connections 2024-03-24T00:36:35.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:33.593+0000 7f7a26000700 1 -- 172.21.15.123:0/4027636768 wait complete. 2024-03-24T00:36:35.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.684+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f7df41abd40 con 0x7f7df4107080 2024-03-24T00:36:35.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.686+0000 7f7deaa00700 1 -- 172.21.15.123:0/3381287789 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f7dec010040 con 0x7f7df4107080 2024-03-24T00:36:35.686 INFO:teuthology.orchestra.run.smithi123.stdout:352187318300 2024-03-24T00:36:35.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 msgr2=0x7f7dd80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 0x7f7dd80a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7de400b2e0 tx=0x7f7de4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 msgr2=0x7f7df41a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:35.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 0x7f7df41a38c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00efb0 tx=0x7f7dec00c3a0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 shutdown_connections 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7dd80a29a0 0x7f7dd80a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df4107080 0x7f7df41a38c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df406a380 0x7f7df419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 --2- 172.21.15.123:0/3381287789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7df4104b90 0x7f7df419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 >> 172.21.15.123:0/3381287789 conn(0x7f7df4077fc0 msgr2=0x7f7df4103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.689+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 shutdown_connections 2024-03-24T00:36:35.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:35.690+0000 7f7df9200700 1 -- 172.21.15.123:0/3381287789 wait complete. 2024-03-24T00:36:36.403 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:36.403 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3381287789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:36:36.403 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.404 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.404 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.404 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:36 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3381287789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:36:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:36 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3381287789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:36:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:36 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:36.788 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318294 got 352187318300 for osd.11 2024-03-24T00:36:36.788 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:38.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:38 smithi123 ceph-mon[31487]: pgmap v347: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:38.453 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:38 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2656070276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:36:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:38 smithi177 ceph-mon[36034]: pgmap v347: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:38.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:38 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2656070276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:36:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:38 smithi204 ceph-mon[36345]: pgmap v347: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:38 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2656070276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:36:38.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 -- 172.21.15.123:0/2367308045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4102440 msgr2=0x7f39a4102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:38.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 --2- 172.21.15.123:0/2367308045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4102440 0x7f39a4102820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3994009990 tx=0x7f3994009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 -- 172.21.15.123:0/2367308045 shutdown_connections 2024-03-24T00:36:38.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 --2- 172.21.15.123:0/2367308045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4107510 0x7f39a410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 --2- 172.21.15.123:0/2367308045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4102440 0x7f39a4102820 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 --2- 172.21.15.123:0/2367308045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a4106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 -- 172.21.15.123:0/2367308045 >> 172.21.15.123:0/2367308045 conn(0x7f39a40792f0 msgr2=0x7f39a4079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 -- 172.21.15.123:0/2367308045 shutdown_connections 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.452+0000 7f39ac400700 1 -- 172.21.15.123:0/2367308045 wait complete. 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.453+0000 7f39ac400700 1 Processor -- start 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 -- start start 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4102440 0x7f39a419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4112560 con 0x7f39a4102440 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a41123e0 con 0x7f39a4107510 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39ac400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a41126e0 con 0x7f39a4102df0 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39aa400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.454+0000 7f39a9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4102440 0x7f39a419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:40404/0 (socket says 172.21.15.123:40404) 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 -- 172.21.15.123:0/1215279418 learned_addr learned my addr 172.21.15.123:0/1215279418 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39a9000700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 -- 172.21.15.123:0/1215279418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 msgr2=0x7f39a419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 -- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4102440 msgr2=0x7f39a419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4102440 0x7f39a419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 -- 172.21.15.123:0/1215279418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4199000 con 0x7f39a4107510 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39a9000700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39aa400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f399c00efc0 tx=0x7f399c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c0092e0 con 0x7f39a4107510 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a4199290 con 0x7f39a4107510 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f399c004bb0 con 0x7f39a4107510 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.455+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c004280 con 0x7f39a4107510 2024-03-24T00:36:38.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.456+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a41a5500 con 0x7f39a4107510 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.457+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4103910 con 0x7f39a4107510 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.459+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f399c01b030 con 0x7f39a4107510 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.460+0000 7f39a2000700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 0x7f39880a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.460+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f399c0ce580 con 0x7f39a4107510 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.460+0000 7f39a9a00700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 0x7f39880a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.461+0000 7f39a9a00700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 0x7f39880a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3994000c00 tx=0x7f399400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:38.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.466+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f399c022020 con 0x7f39a4107510 2024-03-24T00:36:38.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.693+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f39a402da30 con 0x7f39a4107510 2024-03-24T00:36:38.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.693+0000 7f39a2000700 1 -- 172.21.15.123:0/1215279418 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f399c010040 con 0x7f39a4107510 2024-03-24T00:36:38.693 INFO:teuthology.orchestra.run.smithi123.stdout:292057776166 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 msgr2=0x7f39880a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 0x7f39880a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3994000c00 tx=0x7f399400b260 comp rx=0 tx=0).stop 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 msgr2=0x7f39a4198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f399c00efc0 tx=0x7f399c00c3a0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 shutdown_connections 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f39880a2920 0x7f39880a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a4107510 0x7f39a4198820 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a4102440 0x7f39a419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 --2- 172.21.15.123:0/1215279418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39a4102df0 0x7f39a419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:38.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 >> 172.21.15.123:0/1215279418 conn(0x7f39a40792f0 msgr2=0x7f39a41003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:38.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 shutdown_connections 2024-03-24T00:36:38.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:38.696+0000 7f39ac400700 1 -- 172.21.15.123:0/1215279418 wait complete. 2024-03-24T00:36:39.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:39 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1215279418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:36:39.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:39 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1215279418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:36:39.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:39 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1215279418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:36:39.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 -- 172.21.15.123:0/2124109548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 msgr2=0x7f5fd4105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2124109548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd4105c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009990 tx=0x7f5fc0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 -- 172.21.15.123:0/2124109548 shutdown_connections 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2124109548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4106370 0x7f5fd410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2124109548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4101320 0x7f5fd4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2124109548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd4105c90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.694+0000 7f5fd9200700 1 -- 172.21.15.123:0/2124109548 >> 172.21.15.123:0/2124109548 conn(0x7f5fd40770e0 msgr2=0x7f5fd40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.695+0000 7f5fd9200700 1 -- 172.21.15.123:0/2124109548 shutdown_connections 2024-03-24T00:36:39.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.695+0000 7f5fd9200700 1 -- 172.21.15.123:0/2124109548 wait complete. 2024-03-24T00:36:39.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 Processor -- start 2024-03-24T00:36:39.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 -- start start 2024-03-24T00:36:39.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4106370 0x7f5fd41987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4111380 con 0x7f5fd4106370 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4111200 con 0x7f5fd4101cd0 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.696+0000 7f5fd9200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4111500 con 0x7f5fd4101320 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:38522/0 (socket says 172.21.15.123:38522) 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 -- 172.21.15.123:0/2324496114 learned_addr learned my addr 172.21.15.123:0/2324496114 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd2800700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4106370 0x7f5fd41987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:39.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 -- 172.21.15.123:0/2324496114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 msgr2=0x7f5fd419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:39.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 -- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4106370 msgr2=0x7f5fd41987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:39.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4106370 0x7f5fd41987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1e00700 1 -- 172.21.15.123:0/2324496114 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4198fb0 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.697+0000 7f5fd1400700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fd1e00700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400e9a0 tx=0x7f5fc400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400ca10 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd41992a0 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fc400cbb0 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4018680 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.698+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd41a54b0 con 0x7f5fd4101320 2024-03-24T00:36:39.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.699+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd4102770 con 0x7f5fd4101320 2024-03-24T00:36:39.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.701+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5fc4010430 con 0x7f5fd4101320 2024-03-24T00:36:39.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.702+0000 7f5fcaa00700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 0x7f5fb80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:39.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.703+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5fc40cedf0 con 0x7f5fd4101320 2024-03-24T00:36:39.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.703+0000 7f5fd1400700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 0x7f5fb80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:39.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.703+0000 7f5fd1400700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 0x7f5fb80a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4105e40 tx=0x7f5fc000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:39.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.707+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5fc409bce0 con 0x7f5fd4101320 2024-03-24T00:36:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.936+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5fd402da30 con 0x7f5fd4101320 2024-03-24T00:36:39.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.937+0000 7f5fcaa00700 1 -- 172.21.15.123:0/2324496114 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5fc409b5e0 con 0x7f5fd4101320 2024-03-24T00:36:39.937 INFO:teuthology.orchestra.run.smithi123.stdout:111669149768 2024-03-24T00:36:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 msgr2=0x7f5fb80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 0x7f5fb80a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4105e40 tx=0x7f5fc000b260 comp rx=0 tx=0).stop 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 msgr2=0x7f5fd419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400e9a0 tx=0x7f5fc400ecb0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 shutdown_connections 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5fb80a29a0 0x7f5fb80a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd4106370 0x7f5fd41987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd4101320 0x7f5fd419e650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 --2- 172.21.15.123:0/2324496114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fd4101cd0 0x7f5fd419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 >> 172.21.15.123:0/2324496114 conn(0x7f5fd40770e0 msgr2=0x7f5fd406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 shutdown_connections 2024-03-24T00:36:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:39.940+0000 7f5fd9200700 1 -- 172.21.15.123:0/2324496114 wait complete. 2024-03-24T00:36:40.424 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:40 smithi123 ceph-mon[31487]: pgmap v348: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:40.424 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:40 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2324496114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:36:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:40 smithi177 ceph-mon[36034]: pgmap v348: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:40.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:40 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2324496114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:36:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:40 smithi204 ceph-mon[36345]: pgmap v348: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:40 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2324496114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:36:41.388 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776162 got 292057776166 for osd.9 2024-03-24T00:36:41.389 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 -- 172.21.15.123:0/3630900707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 msgr2=0x7f65b4106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 --2- 172.21.15.123:0/3630900707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b4106e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009990 tx=0x7f65a0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 -- 172.21.15.123:0/3630900707 shutdown_connections 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 --2- 172.21.15.123:0/3630900707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 0x7f65b410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 --2- 172.21.15.123:0/3630900707 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 0x7f65b4102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 --2- 172.21.15.123:0/3630900707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b4106e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 -- 172.21.15.123:0/3630900707 >> 172.21.15.123:0/3630900707 conn(0x7f65b40792f0 msgr2=0x7f65b4079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 -- 172.21.15.123:0/3630900707 shutdown_connections 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.976+0000 7f65b8e00700 1 -- 172.21.15.123:0/3630900707 wait complete. 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.977+0000 7f65b8e00700 1 Processor -- start 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.977+0000 7f65b8e00700 1 -- start start 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 0x7f65b41a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 0x7f65b419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4112550 con 0x7f65b4102440 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41123d0 con 0x7f65b4102df0 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b8e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41126d0 con 0x7f65b4107510 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:40388/0 (socket says 172.21.15.123:40388) 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1400700 1 -- 172.21.15.123:0/2656070276 learned_addr learned my addr 172.21.15.123:0/2656070276 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 0x7f65b41a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:42.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b2800700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 0x7f65b419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1400700 1 -- 172.21.15.123:0/2656070276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 msgr2=0x7f65b419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.978+0000 7f65b1400700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 0x7f65b419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b1400700 1 -- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 msgr2=0x7f65b41a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b1400700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 0x7f65b41a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b1400700 1 -- 172.21.15.123:0/2656070276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b419b1b0 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b1400700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f65a000b2e0 tx=0x7f65a00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0026070 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65a000b6c0 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b419b440 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a00173f0 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.979+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b41a7760 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.980+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4068e50 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.982+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f65a0029070 con 0x7f65b4102df0 2024-03-24T00:36:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.984+0000 7f65aaa00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 0x7f65980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.984+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f65a00cf870 con 0x7f65b4102df0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.984+0000 7f65b1e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 0x7f65980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.985+0000 7f65b1e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 0x7f65980a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65b419bb90 tx=0x7f65a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:36.989+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65a0013070 con 0x7f65b4102df0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.218+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f65b41a7930 con 0x7f65b4102df0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.219+0000 7f65aaa00700 1 -- 172.21.15.123:0/2656070276 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f65a009c7f0 con 0x7f65b4102df0 2024-03-24T00:36:42.261 INFO:teuthology.orchestra.run.smithi123.stdout:201863462965 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 msgr2=0x7f65980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 0x7f65980a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65b419bb90 tx=0x7f65a4009260 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 msgr2=0x7f65b41a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f65a000b2e0 tx=0x7f65a00047b0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 shutdown_connections 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65980a29a0 0x7f65980a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f65b4107510 0x7f65b419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4102440 0x7f65b41a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 --2- 172.21.15.123:0/2656070276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b4102df0 0x7f65b41a0dd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 >> 172.21.15.123:0/2656070276 conn(0x7f65b40792f0 msgr2=0x7f65b410d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 shutdown_connections 2024-03-24T00:36:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:37.222+0000 7f65b8e00700 1 -- 172.21.15.123:0/2656070276 wait complete. 2024-03-24T00:36:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:42 smithi177 ceph-mon[36034]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/814348943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:36:42.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3656282071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:36:42.545 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:42 smithi123 ceph-mon[31487]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:42.545 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/814348943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:36:42.545 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3656282071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:36:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:42 smithi204 ceph-mon[36345]: pgmap v349: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:36:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/814348943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:36:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3656282071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:36:42.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-03-24T00:36:42.836 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:43.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:43 smithi177 ceph-mon[36034]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:36:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:43 smithi204 ceph-mon[36345]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:36:43.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:43 smithi123 ceph-mon[31487]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:44.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462965 for osd.6 2024-03-24T00:36:44.075 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:44.270 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:44 smithi123 ceph-mon[31487]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 -- 172.21.15.123:0/3382334260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974108290 msgr2=0x7f6974108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 --2- 172.21.15.123:0/3382334260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974108290 0x7f6974108670 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6964009990 tx=0x7f6964009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 -- 172.21.15.123:0/3382334260 shutdown_connections 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 --2- 172.21.15.123:0/3382334260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 0x7f697410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 --2- 172.21.15.123:0/3382334260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974108290 0x7f6974108670 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 --2- 172.21.15.123:0/3382334260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69741021e0 0x7f6974102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.421+0000 7f697a400700 1 -- 172.21.15.123:0/3382334260 >> 172.21.15.123:0/3382334260 conn(0x7f69740792c0 msgr2=0x7f69740796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.422+0000 7f697a400700 1 -- 172.21.15.123:0/3382334260 shutdown_connections 2024-03-24T00:36:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.422+0000 7f697a400700 1 -- 172.21.15.123:0/3382334260 wait complete. 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 Processor -- start 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 -- start start 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 0x7f69740749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6974108290 0x7f6974074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69741124a0 con 0x7f6974108290 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6974112320 con 0x7f69741021e0 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f697a400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6974112620 con 0x7f6974102ba0 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.423+0000 7f6973400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:40490/0 (socket says 172.21.15.123:40490) 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 -- 172.21.15.123:0/4266922754 learned_addr learned my addr 172.21.15.123:0/4266922754 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6972a00700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 0x7f69740749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973e00700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6974108290 0x7f6974074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 -- 172.21.15.123:0/4266922754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 msgr2=0x7f69740749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 0x7f69740749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 -- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6974108290 msgr2=0x7f6974074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6974108290 0x7f6974074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 -- 172.21.15.123:0/4266922754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69741a6e60 con 0x7f69741021e0 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f6973400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f696400bae0 tx=0x7f696400bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964026070 con 0x7f69741021e0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6964003d70 con 0x7f69741021e0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.424+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69741a70f0 con 0x7f69741021e0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.425+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964021410 con 0x7f69741021e0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.425+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69741a7620 con 0x7f69741021e0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.426+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697410a5e0 con 0x7f69741021e0 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.427+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6964029070 con 0x7f69741021e0 2024-03-24T00:36:44.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.429+0000 7f696be00700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 0x7f69580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.429+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69640cf8e0 con 0x7f69741021e0 2024-03-24T00:36:44.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.429+0000 7f6972a00700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 0x7f69580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.430+0000 7f6972a00700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 0x7f69580a4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6974079a00 tx=0x7f6960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.433+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f696405d740 con 0x7f69741021e0 2024-03-24T00:36:44.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:44 smithi177 ceph-mon[36034]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T00:36:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:44 smithi204 ceph-mon[36345]: pgmap v350: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.678+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f6974109fc0 con 0x7f69741021e0 2024-03-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.678+0000 7f696be00700 1 -- 172.21.15.123:0/4266922754 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6964013070 con 0x7f69741021e0 2024-03-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi123.stdout:257698037804 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 msgr2=0x7f69580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 0x7f69580a4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6974079a00 tx=0x7f6960009260 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 msgr2=0x7f6974074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f696400bae0 tx=0x7f696400bb10 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 shutdown_connections 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69580a2910 0x7f69580a4dd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6974108290 0x7f6974074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69741021e0 0x7f6974074490 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 --2- 172.21.15.123:0/4266922754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6974102ba0 0x7f69740749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 >> 172.21.15.123:0/4266922754 conn(0x7f69740792c0 msgr2=0x7f69741010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 shutdown_connections 2024-03-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:44.683+0000 7f697a400700 1 -- 172.21.15.123:0/4266922754 wait complete. 2024-03-24T00:36:44.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.227+0000 7faf8ec00700 1 -- 172.21.15.123:0/448124216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 msgr2=0x7faf8806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.227+0000 7faf8ec00700 1 --2- 172.21.15.123:0/448124216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf8806a2a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faf74009990 tx=0x7faf74009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 -- 172.21.15.123:0/448124216 shutdown_connections 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 --2- 172.21.15.123:0/448124216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf8810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 --2- 172.21.15.123:0/448124216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf881085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 --2- 172.21.15.123:0/448124216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf8806a2a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 -- 172.21.15.123:0/448124216 >> 172.21.15.123:0/448124216 conn(0x7faf88077fc0 msgr2=0x7faf880783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 -- 172.21.15.123:0/448124216 shutdown_connections 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.228+0000 7faf8ec00700 1 -- 172.21.15.123:0/448124216 wait complete. 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.229+0000 7faf8ec00700 1 Processor -- start 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.229+0000 7faf8ec00700 1 -- start start 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf881016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf88101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88112490 con 0x7faf8806a7e0 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88112310 con 0x7faf88069e40 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf8ec00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88112610 con 0x7faf881081e0 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf87e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf881016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf87e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:40444/0 (socket says 172.21.15.123:40444) 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.230+0000 7faf87e00700 1 -- 172.21.15.123:0/814348943 learned_addr learned my addr 172.21.15.123:0/814348943 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf8cc00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf88101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 -- 172.21.15.123:0/814348943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 msgr2=0x7faf88101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf88101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 -- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 msgr2=0x7faf881016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf881016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 -- 172.21.15.123:0/814348943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf881a6ef0 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf8cc00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf88101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87400700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf881016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf87e00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf7800e9a0 tx=0x7faf7800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7800cbd0 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf7800cd70 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78018750 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf881a71e0 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.231+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf881a7770 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.232+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf88068e50 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.234+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7faf78010430 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.235+0000 7faf84c00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 0x7faf6c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.235+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faf780cedc0 con 0x7faf88069e40 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.236+0000 7faf87400700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 0x7faf6c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.236+0000 7faf87400700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 0x7faf6c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf88078670 tx=0x7faf7400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.239+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf7809bcb0 con 0x7faf88069e40 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.481+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7faf8802da30 con 0x7faf88069e40 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.482+0000 7faf84c00700 1 -- 172.21.15.123:0/814348943 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7faf7809b5b0 con 0x7faf88069e40 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stdout:317827579937 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 msgr2=0x7faf6c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 0x7faf6c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf88078670 tx=0x7faf7400b260 comp rx=0 tx=0).stop 2024-03-24T00:36:44.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 msgr2=0x7faf88101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf7800e9a0 tx=0x7faf7800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 shutdown_connections 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faf6c0a29a0 0x7faf6c0a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf881081e0 0x7faf88101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf88069e40 0x7faf88101160 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 --2- 172.21.15.123:0/814348943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8806a7e0 0x7faf881016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 >> 172.21.15.123:0/814348943 conn(0x7faf88077fc0 msgr2=0x7faf880ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 shutdown_connections 2024-03-24T00:36:44.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.485+0000 7faf8ec00700 1 -- 172.21.15.123:0/814348943 wait complete. 2024-03-24T00:36:45.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 -- 172.21.15.123:0/796875916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c101210 msgr2=0x7f899c10d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 --2- 172.21.15.123:0/796875916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c101210 0x7f899c10d720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8994009f90 tx=0x7f899400b330 comp rx=0 tx=0).stop 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 -- 172.21.15.123:0/796875916 shutdown_connections 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 --2- 172.21.15.123:0/796875916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c101210 0x7f899c10d720 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 --2- 172.21.15.123:0/796875916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c106850 0x7f899c106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 --2- 172.21.15.123:0/796875916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 0x7f899c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 -- 172.21.15.123:0/796875916 >> 172.21.15.123:0/796875916 conn(0x7f899c077690 msgr2=0x7f899c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 -- 172.21.15.123:0/796875916 shutdown_connections 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.601+0000 7f89a2800700 1 -- 172.21.15.123:0/796875916 wait complete. 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.602+0000 7f89a2800700 1 Processor -- start 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 -- start start 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 0x7f899c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c101210 0x7f899c199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c111490 con 0x7f899c100850 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c111310 con 0x7f899c106850 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f89a2800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c111610 con 0x7f899c101210 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f899be00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f899b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 0x7f899c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f899be00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:40466/0 (socket says 172.21.15.123:40466) 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f899be00700 1 -- 172.21.15.123:0/3656282071 learned_addr learned my addr 172.21.15.123:0/3656282071 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.603+0000 7f899aa00700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c101210 0x7f899c199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 -- 172.21.15.123:0/3656282071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c101210 msgr2=0x7f899c199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c101210 0x7f899c199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 -- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 msgr2=0x7f899c199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 0x7f899c199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 -- 172.21.15.123:0/3656282071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c1a4e10 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f899be00700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8994009550 tx=0x7f899400cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994021070 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8994007ac0 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c1a4fe0 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89940052a0 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.604+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c1a54e0 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.605+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c108de0 con 0x7f899c106850 2024-03-24T00:36:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.607+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8994024070 con 0x7f899c106850 2024-03-24T00:36:45.189 INFO:teuthology.orchestra.run.smithi123.stdout:167503724604 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.608+0000 7f8993e00700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 0x7f89800a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.609+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f899400f020 con 0x7f899c106850 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.609+0000 7f899b400700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 0x7f89800a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.609+0000 7f899b400700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 0x7f89800a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f898c009be0 tx=0x7f898c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.612+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f899402b020 con 0x7f899c106850 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.837+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f899c10e6f0 con 0x7f899c106850 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.837+0000 7f8993e00700 1 -- 172.21.15.123:0/3656282071 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f899409cf50 con 0x7f899c106850 2024-03-24T00:36:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 msgr2=0x7f89800a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 0x7f89800a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f898c009be0 tx=0x7f898c009260 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 msgr2=0x7f899c1a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8994009550 tx=0x7f899400cfa0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 shutdown_connections 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89800a29a0 0x7f89800a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f899c106850 0x7f899c1a48d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f899c100850 0x7f899c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 --2- 172.21.15.123:0/3656282071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f899c101210 0x7f899c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.840+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 >> 172.21.15.123:0/3656282071 conn(0x7f899c077690 msgr2=0x7f899c103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.841+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 shutdown_connections 2024-03-24T00:36:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:41.841+0000 7f89a2800700 1 -- 172.21.15.123:0/3656282071 wait complete. 2024-03-24T00:36:45.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:45 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4266922754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:36:45.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:45 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4266922754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:36:45.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:45 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4266922754' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:36:46.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:46 smithi177 ceph-mon[36034]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:46.532 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:46 smithi123 ceph-mon[31487]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:46 smithi204 ceph-mon[36345]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:47.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724604 for osd.5 2024-03-24T00:36:47.153 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:47.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 -- 172.21.15.123:0/1146297793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 msgr2=0x7f6fc8108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 --2- 172.21.15.123:0/1146297793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc8108ee0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009990 tx=0x7f6fb4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 -- 172.21.15.123:0/1146297793 shutdown_connections 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 --2- 172.21.15.123:0/1146297793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81095c0 0x7f6fc8111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 --2- 172.21.15.123:0/1146297793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81044e0 0x7f6fc81048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 --2- 172.21.15.123:0/1146297793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc8108ee0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 -- 172.21.15.123:0/1146297793 >> 172.21.15.123:0/1146297793 conn(0x7f6fc80ffdb0 msgr2=0x7f6fc81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 -- 172.21.15.123:0/1146297793 shutdown_connections 2024-03-24T00:36:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.658+0000 7f6fce600700 1 -- 172.21.15.123:0/1146297793 wait complete. 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.659+0000 7f6fce600700 1 Processor -- start 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 -- start start 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81044e0 0x7f6fc81a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81095c0 0x7f6fc819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81145a0 con 0x7f6fc81095c0 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8114420 con 0x7f6fc8104e90 2024-03-24T00:36:47.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fce600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8114720 con 0x7f6fc81044e0 2024-03-24T00:36:47.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fc7e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fc7e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:55896/0 (socket says 172.21.15.123:55896) 2024-03-24T00:36:47.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fcdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81095c0 0x7f6fc819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fc7e00700 1 -- 172.21.15.123:0/978426158 learned_addr learned my addr 172.21.15.123:0/978426158 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.660+0000 7f6fcd200700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81044e0 0x7f6fc81a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 -- 172.21.15.123:0/978426158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81044e0 msgr2=0x7f6fc81a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81044e0 0x7f6fc81a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 -- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81095c0 msgr2=0x7f6fc819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81095c0 0x7f6fc819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 -- 172.21.15.123:0/978426158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc819afe0 con 0x7f6fc8104e90 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc7e00700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4000c00 tx=0x7f6fb400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4027070 con 0x7f6fc8104e90 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fb4022470 con 0x7f6fc8104e90 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.661+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4017650 con 0x7f6fc8104e90 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.662+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc819b270 con 0x7f6fc8104e90 2024-03-24T00:36:47.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.662+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc81a74e0 con 0x7f6fc8104e90 2024-03-24T00:36:47.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.663+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc81059c0 con 0x7f6fc8104e90 2024-03-24T00:36:47.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.664+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6fb402a070 con 0x7f6fc8104e90 2024-03-24T00:36:47.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.666+0000 7f6fc5600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 0x7f6fb00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.666+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6fb40cf850 con 0x7f6fc8104e90 2024-03-24T00:36:47.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.666+0000 7f6fcd200700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 0x7f6fb00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.667+0000 7f6fcd200700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 0x7f6fb00a4dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009680 tx=0x7f6fbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.670+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6fb405e740 con 0x7f6fc8104e90 2024-03-24T00:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.896+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6fc8068e50 con 0x7f6fc8104e90 2024-03-24T00:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.897+0000 7f6fc5600700 1 -- 172.21.15.123:0/978426158 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6fb4013070 con 0x7f6fc8104e90 2024-03-24T00:36:47.897 INFO:teuthology.orchestra.run.smithi123.stdout:146028888131 2024-03-24T00:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 msgr2=0x7f6fb00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 0x7f6fb00a4dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009680 tx=0x7f6fbc009260 comp rx=0 tx=0).stop 2024-03-24T00:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 msgr2=0x7f6fc81a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4000c00 tx=0x7f6fb400b840 comp rx=0 tx=0).stop 2024-03-24T00:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 shutdown_connections 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6fb00a2910 0x7f6fb00a4dd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc81095c0 0x7f6fc819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fc81044e0 0x7f6fc81a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 --2- 172.21.15.123:0/978426158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc8104e90 0x7f6fc81a0bc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 >> 172.21.15.123:0/978426158 conn(0x7f6fc80ffdb0 msgr2=0x7f6fc81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 shutdown_connections 2024-03-24T00:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:47.900+0000 7f6fce600700 1 -- 172.21.15.123:0/978426158 wait complete. 2024-03-24T00:36:47.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149764 got 111669149768 for osd.3 2024-03-24T00:36:47.922 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:48.319 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:48 smithi123 ceph-mon[31487]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:48.320 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:48 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/978426158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:36:48.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 -- 172.21.15.123:0/2419783460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e4101090 msgr2=0x7fe2e41014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 --2- 172.21.15.123:0/2419783460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e4101090 0x7fe2e41014f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009990 tx=0x7fe2d0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 -- 172.21.15.123:0/2419783460 shutdown_connections 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 --2- 172.21.15.123:0/2419783460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101a30 0x7fe2e410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 --2- 172.21.15.123:0/2419783460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e41070d0 0x7fe2e41074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 --2- 172.21.15.123:0/2419783460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e4101090 0x7fe2e41014f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 -- 172.21.15.123:0/2419783460 >> 172.21.15.123:0/2419783460 conn(0x7fe2e4077fc0 msgr2=0x7fe2e40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 -- 172.21.15.123:0/2419783460 shutdown_connections 2024-03-24T00:36:48.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.486+0000 7fe2eac00700 1 -- 172.21.15.123:0/2419783460 wait complete. 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.487+0000 7fe2eac00700 1 Processor -- start 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.487+0000 7fe2eac00700 1 -- start start 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e41070d0 0x7fe2e419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e4111350 con 0x7fe2e41070d0 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e41111d0 con 0x7fe2e4101a30 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2eac00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e41114d0 con 0x7fe2e4101090 2024-03-24T00:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e41070d0 0x7fe2e419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:55914/0 (socket says 172.21.15.123:55914) 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:52816/0 (socket says 172.21.15.123:52816) 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3400700 1 -- 172.21.15.123:0/3848694233 learned_addr learned my addr 172.21.15.123:0/3848694233 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 -- 172.21.15.123:0/3848694233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 msgr2=0x7fe2e41a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 -- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e41070d0 msgr2=0x7fe2e419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e41070d0 0x7fe2e419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.488+0000 7fe2e3e00700 1 -- 172.21.15.123:0/3848694233 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e419b060 con 0x7fe2e4101090 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2e3400700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2e3e00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400e9f0 tx=0x7fe2d400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d400cb70 con 0x7fe2e4101090 2024-03-24T00:36:48.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2d400ee90 con 0x7fe2e4101090 2024-03-24T00:36:48.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e419b350 con 0x7fe2e4101090 2024-03-24T00:36:48.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4003dc0 con 0x7fe2e4101090 2024-03-24T00:36:48.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.489+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e41a7610 con 0x7fe2e4101090 2024-03-24T00:36:48.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.490+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e4068e50 con 0x7fe2e4101090 2024-03-24T00:36:48.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.492+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2d4004000 con 0x7fe2e4101090 2024-03-24T00:36:48.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.493+0000 7fe2e0c00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 0x7fe2c80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.493+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe2d40ceb00 con 0x7fe2e4101090 2024-03-24T00:36:48.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.494+0000 7fe2e3400700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 0x7fe2c80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.494+0000 7fe2e3400700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 0x7fe2c80a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4101000 tx=0x7fe2d0009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.498+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2d409bb50 con 0x7fe2e4101090 2024-03-24T00:36:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:48 smithi204 ceph-mon[36345]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:48.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:48 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/978426158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:36:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:48 smithi177 ceph-mon[36034]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:48 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/978426158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:36:48.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.728+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe2e419be30 con 0x7fe2e4101090 2024-03-24T00:36:48.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.729+0000 7fe2e0c00700 1 -- 172.21.15.123:0/3848694233 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2d409b450 con 0x7fe2e4101090 2024-03-24T00:36:48.729 INFO:teuthology.orchestra.run.smithi123.stdout:34359738456 2024-03-24T00:36:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 msgr2=0x7fe2c80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 0x7fe2c80a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4101000 tx=0x7fe2d0009590 comp rx=0 tx=0).stop 2024-03-24T00:36:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 msgr2=0x7fe2e41a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400e9f0 tx=0x7fe2d400ed00 comp rx=0 tx=0).stop 2024-03-24T00:36:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 shutdown_connections 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2c80a2910 0x7fe2c80a4dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2e41070d0 0x7fe2e419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2e4101090 0x7fe2e41a0740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 --2- 172.21.15.123:0/3848694233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2e4101a30 0x7fe2e41a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 >> 172.21.15.123:0/3848694233 conn(0x7fe2e4077fc0 msgr2=0x7fe2e4069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.732+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 shutdown_connections 2024-03-24T00:36:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:48.733+0000 7fe2eac00700 1 -- 172.21.15.123:0/3848694233 wait complete. 2024-03-24T00:36:49.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:49 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3848694233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:36:49.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888131 for osd.4 2024-03-24T00:36:49.516 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:49.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:49 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3848694233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:36:49.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:49 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3848694233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:36:49.915 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579932 got 317827579937 for osd.10 2024-03-24T00:36:49.915 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:50.415 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037804 for osd.8 2024-03-24T00:36:50.415 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:50.565 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:50 smithi123 ceph-mon[31487]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:50 smithi204 ceph-mon[36345]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:50.584 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:50 smithi177 ceph-mon[36034]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:51.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738456 for osd.0 2024-03-24T00:36:51.182 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:52 smithi204 ceph-mon[36345]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:52 smithi123 ceph-mon[31487]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:52 smithi177 ceph-mon[36034]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:53.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 -- 172.21.15.123:0/2750903663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d201096c0 msgr2=0x7f4d20111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 --2- 172.21.15.123:0/2750903663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d201096c0 0x7f4d20111290 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b0d0 tx=0x7f4d1800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 -- 172.21.15.123:0/2750903663 shutdown_connections 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 --2- 172.21.15.123:0/2750903663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d201096c0 0x7f4d20111290 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 --2- 172.21.15.123:0/2750903663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201045e0 0x7f4d201049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 --2- 172.21.15.123:0/2750903663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d20104f90 0x7f4d20108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 -- 172.21.15.123:0/2750903663 >> 172.21.15.123:0/2750903663 conn(0x7f4d200ffe70 msgr2=0x7f4d20102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 -- 172.21.15.123:0/2750903663 shutdown_connections 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.025+0000 7f4d28600700 1 -- 172.21.15.123:0/2750903663 wait complete. 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.026+0000 7f4d28600700 1 Processor -- start 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 -- start start 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d201045e0 0x7f4d201a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 0x7f4d201a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 0x7f4d2019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20114700 con 0x7f4d20104f90 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20114580 con 0x7f4d201096c0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d28600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20114880 con 0x7f4d201045e0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.027+0000 7f4d25200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 0x7f4d201a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 0x7f4d2019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d25200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 0x7f4d201a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39284/0 (socket says 172.21.15.123:39284) 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d25200700 1 -- 172.21.15.123:0/1258835205 learned_addr learned my addr 172.21.15.123:0/1258835205 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d25c00700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d201045e0 0x7f4d201a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 -- 172.21.15.123:0/1258835205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d201045e0 msgr2=0x7f4d201a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d201045e0 0x7f4d201a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 -- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 msgr2=0x7f4d201a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 0x7f4d201a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 -- 172.21.15.123:0/1258835205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2019b040 con 0x7f4d201096c0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d26600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 0x7f4d2019a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b770 tx=0x7f4d180079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18003a00 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d18003ba0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1801b770 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.028+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2019b2d0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.029+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d201a75f0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.030+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d20105ac0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.031+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4d18026400 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.033+0000 7f4d1e000700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 0x7f4d080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.033+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4d180cfdb0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.033+0000 7f4d25c00700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 0x7f4d080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.034+0000 7f4d25c00700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 0x7f4d080a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4d2019ba20 tx=0x7f4d14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.037+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d1809ce90 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.261+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4d201a77c0 con 0x7f4d201096c0 2024-03-24T00:36:53.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.262+0000 7f4d1e000700 1 -- 172.21.15.123:0/1258835205 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d1809c790 con 0x7f4d201096c0 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stdout:55834574933 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.266+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 msgr2=0x7f4d080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.266+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 0x7f4d080a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4d2019ba20 tx=0x7f4d14009260 comp rx=0 tx=0).stop 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.266+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 msgr2=0x7f4d2019a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.266+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 0x7f4d2019a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b770 tx=0x7f4d180079a0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 shutdown_connections 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d080a2910 0x7f4d080a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d201096c0 0x7f4d2019a8e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d201045e0 0x7f4d201a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 --2- 172.21.15.123:0/1258835205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20104f90 0x7f4d201a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 >> 172.21.15.123:0/1258835205 conn(0x7f4d200ffe70 msgr2=0x7f4d201008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 shutdown_connections 2024-03-24T00:36:53.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:53.267+0000 7f4d28600700 1 -- 172.21.15.123:0/1258835205 wait complete. 2024-03-24T00:36:54.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:54 smithi123 ceph-mon[31487]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:54.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1258835205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:36:54.380 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574933 for osd.1 2024-03-24T00:36:54.381 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:54 smithi204 ceph-mon[36345]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1258835205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:36:54.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 -- 172.21.15.123:0/3438378113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281021e0 msgr2=0x7f4d28102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3438378113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281021e0 0x7f4d28102660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d140099c0 tx=0x7f4d14009cd0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 -- 172.21.15.123:0/3438378113 shutdown_connections 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3438378113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d28102ba0 0x7f4d2810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3438378113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d281081c0 0x7f4d281085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3438378113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281021e0 0x7f4d28102660 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 -- 172.21.15.123:0/3438378113 >> 172.21.15.123:0/3438378113 conn(0x7f4d280792c0 msgr2=0x7f4d280796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 -- 172.21.15.123:0/3438378113 shutdown_connections 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.637+0000 7f4d2e200700 1 -- 172.21.15.123:0/3438378113 wait complete. 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.638+0000 7f4d2e200700 1 Processor -- start 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 -- start start 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 0x7f4d28074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d28102ba0 0x7f4d280749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d281124c0 con 0x7f4d28102ba0 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28112340 con 0x7f4d281081c0 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d2e200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28112640 con 0x7f4d281021e0 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.639+0000 7f4d27e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d26a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d28102ba0 0x7f4d280749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:55944/0 (socket says 172.21.15.123:55944) 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 -- 172.21.15.123:0/3414621030 learned_addr learned my addr 172.21.15.123:0/3414621030 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27400700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 0x7f4d28074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 -- 172.21.15.123:0/3414621030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 msgr2=0x7f4d28074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 0x7f4d28074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 -- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d28102ba0 msgr2=0x7f4d280749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d28102ba0 0x7f4d280749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 -- 172.21.15.123:0/3414621030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d281a6df0 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27400700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 0x7f4d28074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d27e00700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000ef30 tx=0x7f4d2000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.640+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200092e0 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.641+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d20004bb0 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.641+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20004280 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.641+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d281a7080 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.641+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d281a7540 con 0x7f4d281081c0 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.642+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d28068e50 con 0x7f4d281081c0 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.644+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4d2001b030 con 0x7f4d281081c0 2024-03-24T00:36:54.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.645+0000 7f4d1fe00700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 0x7f4d0c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.645+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4d200ce580 con 0x7f4d281081c0 2024-03-24T00:36:54.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.646+0000 7f4d27400700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 0x7f4d0c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.646+0000 7f4d27400700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 0x7f4d0c0a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c90 tx=0x7f4d18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.654+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d20022020 con 0x7f4d281081c0 2024-03-24T00:36:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:54 smithi177 ceph-mon[36034]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1258835205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:36:54.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.939+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4d2802dc20 con 0x7f4d281081c0 2024-03-24T00:36:54.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.940+0000 7f4d1fe00700 1 -- 172.21.15.123:0/3414621030 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4d20010040 con 0x7f4d281081c0 2024-03-24T00:36:54.941 INFO:teuthology.orchestra.run.smithi123.stdout:223338299444 2024-03-24T00:36:54.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 msgr2=0x7f4d0c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 0x7f4d0c0a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c90 tx=0x7f4d18009260 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 msgr2=0x7f4d28074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000ef30 tx=0x7f4d2000c3a0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 shutdown_connections 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4d0c0a2910 0x7f4d0c0a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d281081c0 0x7f4d28074f10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d281021e0 0x7f4d28074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 --2- 172.21.15.123:0/3414621030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d28102ba0 0x7f4d280749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.943+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 >> 172.21.15.123:0/3414621030 conn(0x7f4d280792c0 msgr2=0x7f4d28100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.944+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 shutdown_connections 2024-03-24T00:36:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:54.944+0000 7f4d2e200700 1 -- 172.21.15.123:0/3414621030 wait complete. 2024-03-24T00:36:55.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3414621030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:36:55.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3414621030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:36:55.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3414621030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:36:55.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299439 got 223338299444 for osd.7 2024-03-24T00:36:55.825 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:55.825 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T00:36:55.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:36:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:36:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:56 smithi204 ceph-mon[36345]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:56 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:56.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:56 smithi123 ceph-mon[31487]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:56.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:56.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:56 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:56 smithi177 ceph-mon[36034]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:36:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:56 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:57.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:57.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:57.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:57.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:57.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:57.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 -- 172.21.15.123:0/1221799298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc0766c0 msgr2=0x7fa1fc076aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 --2- 172.21.15.123:0/1221799298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc076aa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009990 tx=0x7fa1ec009ca0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 -- 172.21.15.123:0/1221799298 shutdown_connections 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 --2- 172.21.15.123:0/1221799298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 0x7fa1fc115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 --2- 172.21.15.123:0/1221799298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc076aa0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 --2- 172.21.15.123:0/1221799298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc112b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 -- 172.21.15.123:0/1221799298 >> 172.21.15.123:0/1221799298 conn(0x7fa1fc0ffe70 msgr2=0x7fa1fc102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 -- 172.21.15.123:0/1221799298 shutdown_connections 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.482+0000 7fa202c00700 1 -- 172.21.15.123:0/1221799298 wait complete. 2024-03-24T00:36:58.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.483+0000 7fa202c00700 1 Processor -- start 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 -- start start 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 0x7fa1fc1a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc118b50 con 0x7fa1fc0766c0 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1189d0 con 0x7fa1fc076fe0 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa202c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc118cd0 con 0x7fa1fc113040 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa1fb400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa1fbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa1fb400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:42362/0 (socket says 172.21.15.123:42362) 2024-03-24T00:36:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa1fb400700 1 -- 172.21.15.123:0/3579645218 learned_addr learned my addr 172.21.15.123:0/3579645218 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:36:58.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.484+0000 7fa200c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 0x7fa1fc1a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 -- 172.21.15.123:0/3579645218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 msgr2=0x7fa1fc1a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 0x7fa1fc1a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 -- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc0766c0 msgr2=0x7fa1fc19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 -- 172.21.15.123:0/3579645218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc1a4030 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1fb400700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800e9a0 tx=0x7fa1e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e800cbd0 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e800cd70 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc1a4320 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8018750 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.485+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc1abba0 con 0x7fa1fc076fe0 2024-03-24T00:36:58.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.486+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc068e50 con 0x7fa1fc076fe0 2024-03-24T00:36:58.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.488+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa1e8010430 con 0x7fa1fc076fe0 2024-03-24T00:36:58.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.489+0000 7fa1f8c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 0x7fa1e00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.489+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1e80cecf0 con 0x7fa1fc076fe0 2024-03-24T00:36:58.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.489+0000 7fa1fbe00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 0x7fa1e00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.490+0000 7fa1fbe00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 0x7fa1e00a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec006010 tx=0x7fa1ec00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.493+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1e809bd40 con 0x7fa1fc076fe0 2024-03-24T00:36:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:36:58 smithi204 ceph-mon[36345]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:36:58 smithi123 ceph-mon[31487]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:36:58 smithi177 ceph-mon[36034]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.711+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1fc1a0380 con 0x7fa1e00a2910 2024-03-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.723+0000 7fa1f8c00700 1 -- 172.21.15.123:0/3579645218 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258751 (secure 0 0 0) 0x7fa1fc1a0380 con 0x7fa1e00a2910 2024-03-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 msgr2=0x7fa1e00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 0x7fa1e00a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec006010 tx=0x7fa1ec00b260 comp rx=0 tx=0).stop 2024-03-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 msgr2=0x7fa1fc19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800e9a0 tx=0x7fa1e800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 shutdown_connections 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e00a2910 0x7fa1e00a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1fc113040 0x7fa1fc1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1fc0766c0 0x7fa1fc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 --2- 172.21.15.123:0/3579645218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1fc076fe0 0x7fa1fc19f560 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 >> 172.21.15.123:0/3579645218 conn(0x7fa1fc0ffe70 msgr2=0x7fa1fc113cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 shutdown_connections 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:36:58.730+0000 7fa202c00700 1 -- 172.21.15.123:0/3579645218 wait complete. 2024-03-24T00:36:58.733 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:36:59.604 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-24T00:36:57.101769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653545,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":298,"num_read_kb":258,"num_write":267,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":709,"ondisk_log_size":709,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354104,"kb_used_data":13272,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587720,"statfs":{"total":1151940427776,"available":1151577825280,"internally_reserved":0,"allocated":13590528,"data_stored":7551627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005237"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082727+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:36:41.082727+0000","last_peered":"2024-03-24T00:36:41.082727+0000","last_clean":"2024-03-24T00:36:41.082727+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:36:41.082727+0000","last_undegraded":"2024-03-24T00:36:41.082727+0000","last_fullsized":"2024-03-24T00:36:41.082727+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082980+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:36:41.082980+0000","last_peered":"2024-03-24T00:36:41.082980+0000","last_clean":"2024-03-24T00:36:41.082980+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:36:41.082980+0000","last_undegraded":"2024-03-24T00:36:41.082980+0000","last_fullsized":"2024-03-24T00:36:41.082980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082446+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:36:41.082446+0000","last_peered":"2024-03-24T00:36:41.082446+0000","last_clean":"2024-03-24T00:36:41.082446+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:36:41.082446+0000","last_undegraded":"2024-03-24T00:36:41.082446+0000","last_fullsized":"2024-03-24T00:36:41.082446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.058888+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:36:41.058888+0000","last_peered":"2024-03-24T00:36:41.058888+0000","last_clean":"2024-03-24T00:36:41.058888+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:36:41.058888+0000","last_undegraded":"2024-03-24T00:36:41.058888+0000","last_fullsized":"2024-03-24T00:36:41.058888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.068426+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:36:41.068426+0000","last_peered":"2024-03-24T00:36:41.068426+0000","last_clean":"2024-03-24T00:36:41.068426+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:36:41.068426+0000","last_undegraded":"2024-03-24T00:36:41.068426+0000","last_fullsized":"2024-03-24T00:36:41.068426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073848+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:36:41.073848+0000","last_peered":"2024-03-24T00:36:41.073848+0000","last_clean":"2024-03-24T00:36:41.073848+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:36:41.073848+0000","last_undegraded":"2024-03-24T00:36:41.073848+0000","last_fullsized":"2024-03-24T00:36:41.073848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074753+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:36:41.074753+0000","last_peered":"2024-03-24T00:36:41.074753+0000","last_clean":"2024-03-24T00:36:41.074753+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:36:41.074753+0000","last_undegraded":"2024-03-24T00:36:41.074753+0000","last_fullsized":"2024-03-24T00:36:41.074753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065233+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:36:41.065233+0000","last_peered":"2024-03-24T00:36:41.065233+0000","last_clean":"2024-03-24T00:36:41.065233+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:36:41.065233+0000","last_undegraded":"2024-03-24T00:36:41.065233+0000","last_fullsized":"2024-03-24T00:36:41.065233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063340+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:36:41.063340+0000","last_peered":"2024-03-24T00:36:41.063340+0000","last_clean":"2024-03-24T00:36:41.063340+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:36:41.063340+0000","last_undegraded":"2024-03-24T00:36:41.063340+0000","last_fullsized":"2024-03-24T00:36:41.063340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078500+0000","last_change":"2024-03-24T00:35:42.436405+0000","last_active":"2024-03-24T00:36:41.078500+0000","last_peered":"2024-03-24T00:36:41.078500+0000","last_clean":"2024-03-24T00:36:41.078500+0000","last_became_active":"2024-03-24T00:35:42.436022+0000","last_became_peered":"2024-03-24T00:35:42.436022+0000","last_unstale":"2024-03-24T00:36:41.078500+0000","last_undegraded":"2024-03-24T00:36:41.078500+0000","last_fullsized":"2024-03-24T00:36:41.078500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:59.022152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063157+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:36:41.063157+0000","last_peered":"2024-03-24T00:36:41.063157+0000","last_clean":"2024-03-24T00:36:41.063157+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:36:41.063157+0000","last_undegraded":"2024-03-24T00:36:41.063157+0000","last_fullsized":"2024-03-24T00:36:41.063157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.046590+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:36:41.046590+0000","last_peered":"2024-03-24T00:36:41.046590+0000","last_clean":"2024-03-24T00:36:41.046590+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:36:41.046590+0000","last_undegraded":"2024-03-24T00:36:41.046590+0000","last_fullsized":"2024-03-24T00:36:41.046590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.077839+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:36:41.077839+0000","last_peered":"2024-03-24T00:36:41.077839+0000","last_clean":"2024-03-24T00:36:41.077839+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:36:41.077839+0000","last_undegraded":"2024-03-24T00:36:41.077839+0000","last_fullsized":"2024-03-24T00:36:41.077839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.084192+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:36:41.084192+0000","last_peered":"2024-03-24T00:36:41.084192+0000","last_clean":"2024-03-24T00:36:41.084192+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:36:41.084192+0000","last_undegraded":"2024-03-24T00:36:41.084192+0000","last_fullsized":"2024-03-24T00:36:41.084192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083963+0000","last_change":"2024-03-24T00:35:42.433339+0000","last_active":"2024-03-24T00:36:41.083963+0000","last_peered":"2024-03-24T00:36:41.083963+0000","last_clean":"2024-03-24T00:36:41.083963+0000","last_became_active":"2024-03-24T00:35:42.432819+0000","last_became_peered":"2024-03-24T00:35:42.432819+0000","last_unstale":"2024-03-24T00:36:41.083963+0000","last_undegraded":"2024-03-24T00:36:41.083963+0000","last_fullsized":"2024-03-24T00:36:41.083963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:09.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082904+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:36:41.082904+0000","last_peered":"2024-03-24T00:36:41.082904+0000","last_clean":"2024-03-24T00:36:41.082904+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:36:41.082904+0000","last_undegraded":"2024-03-24T00:36:41.082904+0000","last_fullsized":"2024-03-24T00:36:41.082904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082789+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:36:41.082789+0000","last_peered":"2024-03-24T00:36:41.082789+0000","last_clean":"2024-03-24T00:36:41.082789+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:36:41.082789+0000","last_undegraded":"2024-03-24T00:36:41.082789+0000","last_fullsized":"2024-03-24T00:36:41.082789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.079303+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:36:41.079303+0000","last_peered":"2024-03-24T00:36:41.079303+0000","last_clean":"2024-03-24T00:36:41.079303+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:36:41.079303+0000","last_undegraded":"2024-03-24T00:36:41.079303+0000","last_fullsized":"2024-03-24T00:36:41.079303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056499+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:36:41.056499+0000","last_peered":"2024-03-24T00:36:41.056499+0000","last_clean":"2024-03-24T00:36:41.056499+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:36:41.056499+0000","last_undegraded":"2024-03-24T00:36:41.056499+0000","last_fullsized":"2024-03-24T00:36:41.056499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435460+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:42.435460+0000","last_peered":"2024-03-24T00:35:42.435460+0000","last_clean":"2024-03-24T00:35:42.435460+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:42.435460+0000","last_undegraded":"2024-03-24T00:35:42.435460+0000","last_fullsized":"2024-03-24T00:35:42.435460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062812+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:36:41.062812+0000","last_peered":"2024-03-24T00:36:41.062812+0000","last_clean":"2024-03-24T00:36:41.062812+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:36:41.062812+0000","last_undegraded":"2024-03-24T00:36:41.062812+0000","last_fullsized":"2024-03-24T00:36:41.062812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.433298+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:42.433298+0000","last_peered":"2024-03-24T00:35:42.433298+0000","last_clean":"2024-03-24T00:35:42.433298+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:42.433298+0000","last_undegraded":"2024-03-24T00:35:42.433298+0000","last_fullsized":"2024-03-24T00:35:42.433298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080791+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:36:41.080791+0000","last_peered":"2024-03-24T00:36:41.080791+0000","last_clean":"2024-03-24T00:36:41.080791+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:36:41.080791+0000","last_undegraded":"2024-03-24T00:36:41.080791+0000","last_fullsized":"2024-03-24T00:36:41.080791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435536+0000","last_change":"2024-03-24T00:35:42.435006+0000","last_active":"2024-03-24T00:35:42.435536+0000","last_peered":"2024-03-24T00:35:42.435536+0000","last_clean":"2024-03-24T00:35:42.435536+0000","last_became_active":"2024-03-24T00:35:42.429558+0000","last_became_peered":"2024-03-24T00:35:42.429558+0000","last_unstale":"2024-03-24T00:35:42.435536+0000","last_undegraded":"2024-03-24T00:35:42.435536+0000","last_fullsized":"2024-03-24T00:35:42.435536+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:09.212462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061003+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:36:41.061003+0000","last_peered":"2024-03-24T00:36:41.061003+0000","last_clean":"2024-03-24T00:36:41.061003+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:36:41.061003+0000","last_undegraded":"2024-03-24T00:36:41.061003+0000","last_fullsized":"2024-03-24T00:36:41.061003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431452+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:42.431452+0000","last_peered":"2024-03-24T00:35:42.431452+0000","last_clean":"2024-03-24T00:35:42.431452+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:42.431452+0000","last_undegraded":"2024-03-24T00:35:42.431452+0000","last_fullsized":"2024-03-24T00:35:42.431452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431007+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:42.431007+0000","last_peered":"2024-03-24T00:35:42.431007+0000","last_clean":"2024-03-24T00:35:42.431007+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:42.431007+0000","last_undegraded":"2024-03-24T00:35:42.431007+0000","last_fullsized":"2024-03-24T00:35:42.431007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074671+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:36:41.074671+0000","last_peered":"2024-03-24T00:36:41.074671+0000","last_clean":"2024-03-24T00:36:41.074671+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:36:41.074671+0000","last_undegraded":"2024-03-24T00:36:41.074671+0000","last_fullsized":"2024-03-24T00:36:41.074671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422152+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:42.422152+0000","last_peered":"2024-03-24T00:35:42.422152+0000","last_clean":"2024-03-24T00:35:42.422152+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:42.422152+0000","last_undegraded":"2024-03-24T00:35:42.422152+0000","last_fullsized":"2024-03-24T00:35:42.422152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061844+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:36:41.061844+0000","last_peered":"2024-03-24T00:36:41.061844+0000","last_clean":"2024-03-24T00:36:41.061844+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:36:41.061844+0000","last_undegraded":"2024-03-24T00:36:41.061844+0000","last_fullsized":"2024-03-24T00:36:41.061844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422989+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:42.422989+0000","last_peered":"2024-03-24T00:35:42.422989+0000","last_clean":"2024-03-24T00:35:42.422989+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:42.422989+0000","last_undegraded":"2024-03-24T00:35:42.422989+0000","last_fullsized":"2024-03-24T00:35:42.422989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071616+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:36:41.071616+0000","last_peered":"2024-03-24T00:36:41.071616+0000","last_clean":"2024-03-24T00:36:41.071616+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:36:41.071616+0000","last_undegraded":"2024-03-24T00:36:41.071616+0000","last_fullsized":"2024-03-24T00:36:41.071616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430389+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:42.430389+0000","last_peered":"2024-03-24T00:35:42.430389+0000","last_clean":"2024-03-24T00:35:42.430389+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:42.430389+0000","last_undegraded":"2024-03-24T00:35:42.430389+0000","last_fullsized":"2024-03-24T00:35:42.430389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073853+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:36:41.073853+0000","last_peered":"2024-03-24T00:36:41.073853+0000","last_clean":"2024-03-24T00:36:41.073853+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:36:41.073853+0000","last_undegraded":"2024-03-24T00:36:41.073853+0000","last_fullsized":"2024-03-24T00:36:41.073853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422441+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:42.422441+0000","last_peered":"2024-03-24T00:35:42.422441+0000","last_clean":"2024-03-24T00:35:42.422441+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:42.422441+0000","last_undegraded":"2024-03-24T00:35:42.422441+0000","last_fullsized":"2024-03-24T00:35:42.422441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078541+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:36:41.078541+0000","last_peered":"2024-03-24T00:36:41.078541+0000","last_clean":"2024-03-24T00:36:41.078541+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:36:41.078541+0000","last_undegraded":"2024-03-24T00:36:41.078541+0000","last_fullsized":"2024-03-24T00:36:41.078541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423580+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:42.423580+0000","last_peered":"2024-03-24T00:35:42.423580+0000","last_clean":"2024-03-24T00:35:42.423580+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:42.423580+0000","last_undegraded":"2024-03-24T00:35:42.423580+0000","last_fullsized":"2024-03-24T00:35:42.423580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083152+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:36:41.083152+0000","last_peered":"2024-03-24T00:36:41.083152+0000","last_clean":"2024-03-24T00:36:41.083152+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:36:41.083152+0000","last_undegraded":"2024-03-24T00:36:41.083152+0000","last_fullsized":"2024-03-24T00:36:41.083152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431398+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:42.431398+0000","last_peered":"2024-03-24T00:35:42.431398+0000","last_clean":"2024-03-24T00:35:42.431398+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:42.431398+0000","last_undegraded":"2024-03-24T00:35:42.431398+0000","last_fullsized":"2024-03-24T00:35:42.431398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057322+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:36:41.057322+0000","last_peered":"2024-03-24T00:36:41.057322+0000","last_clean":"2024-03-24T00:36:41.057322+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:36:41.057322+0000","last_undegraded":"2024-03-24T00:36:41.057322+0000","last_fullsized":"2024-03-24T00:36:41.057322+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":215,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:54.198638+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:36:54.198638+0000","last_peered":"2024-03-24T00:36:54.198638+0000","last_clean":"2024-03-24T00:36:54.198638+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:36:54.198638+0000","last_undegraded":"2024-03-24T00:36:54.198638+0000","last_fullsized":"2024-03-24T00:36:54.198638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62145,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":167,"num_read_kb":144,"num_write":29,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062432+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:36:41.062432+0000","last_peered":"2024-03-24T00:36:41.062432+0000","last_clean":"2024-03-24T00:36:41.062432+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:36:41.062432+0000","last_undegraded":"2024-03-24T00:36:41.062432+0000","last_fullsized":"2024-03-24T00:36:41.062432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424521+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:42.424521+0000","last_peered":"2024-03-24T00:35:42.424521+0000","last_clean":"2024-03-24T00:35:42.424521+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:42.424521+0000","last_undegraded":"2024-03-24T00:35:42.424521+0000","last_fullsized":"2024-03-24T00:35:42.424521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067324+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:36:41.067324+0000","last_peered":"2024-03-24T00:36:41.067324+0000","last_clean":"2024-03-24T00:36:41.067324+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:36:41.067324+0000","last_undegraded":"2024-03-24T00:36:41.067324+0000","last_fullsized":"2024-03-24T00:36:41.067324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430861+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:42.430861+0000","last_peered":"2024-03-24T00:35:42.430861+0000","last_clean":"2024-03-24T00:35:42.430861+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:42.430861+0000","last_undegraded":"2024-03-24T00:35:42.430861+0000","last_fullsized":"2024-03-24T00:35:42.430861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078222+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:36:41.078222+0000","last_peered":"2024-03-24T00:36:41.078222+0000","last_clean":"2024-03-24T00:36:41.078222+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:36:41.078222+0000","last_undegraded":"2024-03-24T00:36:41.078222+0000","last_fullsized":"2024-03-24T00:36:41.078222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425512+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:42.425512+0000","last_peered":"2024-03-24T00:35:42.425512+0000","last_clean":"2024-03-24T00:35:42.425512+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:42.425512+0000","last_undegraded":"2024-03-24T00:35:42.425512+0000","last_fullsized":"2024-03-24T00:35:42.425512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067274+0000","last_change":"2024-03-24T00:35:42.436171+0000","last_active":"2024-03-24T00:36:41.067274+0000","last_peered":"2024-03-24T00:36:41.067274+0000","last_clean":"2024-03-24T00:36:41.067274+0000","last_became_active":"2024-03-24T00:35:42.435757+0000","last_became_peered":"2024-03-24T00:35:42.435757+0000","last_unstale":"2024-03-24T00:36:41.067274+0000","last_undegraded":"2024-03-24T00:36:41.067274+0000","last_fullsized":"2024-03-24T00:36:41.067274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:08.194563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430606+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:42.430606+0000","last_peered":"2024-03-24T00:35:42.430606+0000","last_clean":"2024-03-24T00:35:42.430606+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:42.430606+0000","last_undegraded":"2024-03-24T00:35:42.430606+0000","last_fullsized":"2024-03-24T00:35:42.430606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059832+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:36:41.059832+0000","last_peered":"2024-03-24T00:36:41.059832+0000","last_clean":"2024-03-24T00:36:41.059832+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:36:41.059832+0000","last_undegraded":"2024-03-24T00:36:41.059832+0000","last_fullsized":"2024-03-24T00:36:41.059832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.438512+0000","last_change":"2024-03-24T00:35:42.438099+0000","last_active":"2024-03-24T00:35:42.438512+0000","last_peered":"2024-03-24T00:35:42.438512+0000","last_clean":"2024-03-24T00:35:42.438512+0000","last_became_active":"2024-03-24T00:35:42.432549+0000","last_became_peered":"2024-03-24T00:35:42.432549+0000","last_unstale":"2024-03-24T00:35:42.438512+0000","last_undegraded":"2024-03-24T00:35:42.438512+0000","last_fullsized":"2024-03-24T00:35:42.438512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:33.579265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.053232+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:36:41.053232+0000","last_peered":"2024-03-24T00:36:41.053232+0000","last_clean":"2024-03-24T00:36:41.053232+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:36:41.053232+0000","last_undegraded":"2024-03-24T00:36:41.053232+0000","last_fullsized":"2024-03-24T00:36:41.053232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.434686+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:42.434686+0000","last_peered":"2024-03-24T00:35:42.434686+0000","last_clean":"2024-03-24T00:35:42.434686+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:42.434686+0000","last_undegraded":"2024-03-24T00:35:42.434686+0000","last_fullsized":"2024-03-24T00:35:42.434686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.049175+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:36:41.049175+0000","last_peered":"2024-03-24T00:36:41.049175+0000","last_clean":"2024-03-24T00:36:41.049175+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:36:41.049175+0000","last_undegraded":"2024-03-24T00:36:41.049175+0000","last_fullsized":"2024-03-24T00:36:41.049175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423519+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:42.423519+0000","last_peered":"2024-03-24T00:35:42.423519+0000","last_clean":"2024-03-24T00:35:42.423519+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:42.423519+0000","last_undegraded":"2024-03-24T00:35:42.423519+0000","last_fullsized":"2024-03-24T00:35:42.423519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082180+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:36:41.082180+0000","last_peered":"2024-03-24T00:36:41.082180+0000","last_clean":"2024-03-24T00:36:41.082180+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:36:41.082180+0000","last_undegraded":"2024-03-24T00:36:41.082180+0000","last_fullsized":"2024-03-24T00:36:41.082180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.051429+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:36:41.051429+0000","last_peered":"2024-03-24T00:36:41.051429+0000","last_clean":"2024-03-24T00:36:41.051429+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:36:41.051429+0000","last_undegraded":"2024-03-24T00:36:41.051429+0000","last_fullsized":"2024-03-24T00:36:41.051429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422884+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:42.422884+0000","last_peered":"2024-03-24T00:35:42.422884+0000","last_clean":"2024-03-24T00:35:42.422884+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:42.422884+0000","last_undegraded":"2024-03-24T00:35:42.422884+0000","last_fullsized":"2024-03-24T00:35:42.422884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063946+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:36:41.063946+0000","last_peered":"2024-03-24T00:36:41.063946+0000","last_clean":"2024-03-24T00:36:41.063946+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:36:41.063946+0000","last_undegraded":"2024-03-24T00:36:41.063946+0000","last_fullsized":"2024-03-24T00:36:41.063946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425608+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:42.425608+0000","last_peered":"2024-03-24T00:35:42.425608+0000","last_clean":"2024-03-24T00:35:42.425608+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:42.425608+0000","last_undegraded":"2024-03-24T00:35:42.425608+0000","last_fullsized":"2024-03-24T00:35:42.425608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423432+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:42.423432+0000","last_peered":"2024-03-24T00:35:42.423432+0000","last_clean":"2024-03-24T00:35:42.423432+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:42.423432+0000","last_undegraded":"2024-03-24T00:35:42.423432+0000","last_fullsized":"2024-03-24T00:35:42.423432+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059155+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:36:41.059155+0000","last_peered":"2024-03-24T00:36:41.059155+0000","last_clean":"2024-03-24T00:36:41.059155+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:36:41.059155+0000","last_undegraded":"2024-03-24T00:36:41.059155+0000","last_fullsized":"2024-03-24T00:36:41.059155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423281+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:42.423281+0000","last_peered":"2024-03-24T00:35:42.423281+0000","last_clean":"2024-03-24T00:35:42.423281+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:42.423281+0000","last_undegraded":"2024-03-24T00:35:42.423281+0000","last_fullsized":"2024-03-24T00:35:42.423281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431457+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:42.431457+0000","last_peered":"2024-03-24T00:35:42.431457+0000","last_clean":"2024-03-24T00:35:42.431457+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:42.431457+0000","last_undegraded":"2024-03-24T00:35:42.431457+0000","last_fullsized":"2024-03-24T00:35:42.431457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074539+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:36:41.074539+0000","last_peered":"2024-03-24T00:36:41.074539+0000","last_clean":"2024-03-24T00:36:41.074539+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:36:41.074539+0000","last_undegraded":"2024-03-24T00:36:41.074539+0000","last_fullsized":"2024-03-24T00:36:41.074539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:46.358125+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:35:46.358125+0000","last_peered":"2024-03-24T00:35:46.358125+0000","last_clean":"2024-03-24T00:35:46.358125+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:35:46.358125+0000","last_undegraded":"2024-03-24T00:35:46.358125+0000","last_fullsized":"2024-03-24T00:35:46.358125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065991+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:36:41.065991+0000","last_peered":"2024-03-24T00:36:41.065991+0000","last_clean":"2024-03-24T00:36:41.065991+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:36:41.065991+0000","last_undegraded":"2024-03-24T00:36:41.065991+0000","last_fullsized":"2024-03-24T00:36:41.065991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425899+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:42.425899+0000","last_peered":"2024-03-24T00:35:42.425899+0000","last_clean":"2024-03-24T00:35:42.425899+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:42.425899+0000","last_undegraded":"2024-03-24T00:35:42.425899+0000","last_fullsized":"2024-03-24T00:35:42.425899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075280+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:36:41.075280+0000","last_peered":"2024-03-24T00:36:41.075280+0000","last_clean":"2024-03-24T00:36:41.075280+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:36:41.075280+0000","last_undegraded":"2024-03-24T00:36:41.075280+0000","last_fullsized":"2024-03-24T00:36:41.075280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.862632+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:42.862632+0000","last_peered":"2024-03-24T00:35:42.862632+0000","last_clean":"2024-03-24T00:35:42.862632+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:42.862632+0000","last_undegraded":"2024-03-24T00:35:42.862632+0000","last_fullsized":"2024-03-24T00:35:42.862632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.024641+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:36:41.024641+0000","last_peered":"2024-03-24T00:36:41.024641+0000","last_clean":"2024-03-24T00:36:41.024641+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:36:41.024641+0000","last_undegraded":"2024-03-24T00:36:41.024641+0000","last_fullsized":"2024-03-24T00:36:41.024641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.428625+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:42.428625+0000","last_peered":"2024-03-24T00:35:42.428625+0000","last_clean":"2024-03-24T00:35:42.428625+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:42.428625+0000","last_undegraded":"2024-03-24T00:35:42.428625+0000","last_fullsized":"2024-03-24T00:35:42.428625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082612+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:36:41.082612+0000","last_peered":"2024-03-24T00:36:41.082612+0000","last_clean":"2024-03-24T00:36:41.082612+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:36:41.082612+0000","last_undegraded":"2024-03-24T00:36:41.082612+0000","last_fullsized":"2024-03-24T00:36:41.082612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424512+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:42.424512+0000","last_peered":"2024-03-24T00:35:42.424512+0000","last_clean":"2024-03-24T00:35:42.424512+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:42.424512+0000","last_undegraded":"2024-03-24T00:35:42.424512+0000","last_fullsized":"2024-03-24T00:35:42.424512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080523+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:36:41.080523+0000","last_peered":"2024-03-24T00:36:41.080523+0000","last_clean":"2024-03-24T00:36:41.080523+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:36:41.080523+0000","last_undegraded":"2024-03-24T00:36:41.080523+0000","last_fullsized":"2024-03-24T00:36:41.080523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430730+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:42.430730+0000","last_peered":"2024-03-24T00:35:42.430730+0000","last_clean":"2024-03-24T00:35:42.430730+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:42.430730+0000","last_undegraded":"2024-03-24T00:35:42.430730+0000","last_fullsized":"2024-03-24T00:35:42.430730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071383+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:36:41.071383+0000","last_peered":"2024-03-24T00:36:41.071383+0000","last_clean":"2024-03-24T00:36:41.071383+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:36:41.071383+0000","last_undegraded":"2024-03-24T00:36:41.071383+0000","last_fullsized":"2024-03-24T00:36:41.071383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.432672+0000","last_change":"2024-03-24T00:35:42.432365+0000","last_active":"2024-03-24T00:35:42.432672+0000","last_peered":"2024-03-24T00:35:42.432672+0000","last_clean":"2024-03-24T00:35:42.432672+0000","last_became_active":"2024-03-24T00:35:42.432113+0000","last_became_peered":"2024-03-24T00:35:42.432113+0000","last_unstale":"2024-03-24T00:35:42.432672+0000","last_undegraded":"2024-03-24T00:35:42.432672+0000","last_fullsized":"2024-03-24T00:35:42.432672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:32.467534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065060+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:36:41.065060+0000","last_peered":"2024-03-24T00:36:41.065060+0000","last_clean":"2024-03-24T00:36:41.065060+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:36:41.065060+0000","last_undegraded":"2024-03-24T00:36:41.065060+0000","last_fullsized":"2024-03-24T00:36:41.065060+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431200+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:42.431200+0000","last_peered":"2024-03-24T00:35:42.431200+0000","last_clean":"2024-03-24T00:35:42.431200+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:42.431200+0000","last_undegraded":"2024-03-24T00:35:42.431200+0000","last_fullsized":"2024-03-24T00:35:42.431200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063211+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:36:41.063211+0000","last_peered":"2024-03-24T00:36:41.063211+0000","last_clean":"2024-03-24T00:36:41.063211+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:36:41.063211+0000","last_undegraded":"2024-03-24T00:36:41.063211+0000","last_fullsized":"2024-03-24T00:36:41.063211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.866804+0000","last_change":"2024-03-24T00:35:42.866804+0000","last_active":"2024-03-24T00:35:42.866804+0000","last_peered":"2024-03-24T00:35:42.866804+0000","last_clean":"2024-03-24T00:35:42.866804+0000","last_became_active":"2024-03-24T00:35:42.863933+0000","last_became_peered":"2024-03-24T00:35:42.863933+0000","last_unstale":"2024-03-24T00:35:42.866804+0000","last_undegraded":"2024-03-24T00:35:42.866804+0000","last_fullsized":"2024-03-24T00:35:42.866804+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:55.475527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062049+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:36:41.062049+0000","last_peered":"2024-03-24T00:36:41.062049+0000","last_clean":"2024-03-24T00:36:41.062049+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:36:41.062049+0000","last_undegraded":"2024-03-24T00:36:41.062049+0000","last_fullsized":"2024-03-24T00:36:41.062049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.052456+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:36:41.052456+0000","last_peered":"2024-03-24T00:36:41.052456+0000","last_clean":"2024-03-24T00:36:41.052456+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:36:41.052456+0000","last_undegraded":"2024-03-24T00:36:41.052456+0000","last_fullsized":"2024-03-24T00:36:41.052456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074257+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:36:41.074257+0000","last_peered":"2024-03-24T00:36:41.074257+0000","last_clean":"2024-03-24T00:36:41.074257+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:36:41.074257+0000","last_undegraded":"2024-03-24T00:36:41.074257+0000","last_fullsized":"2024-03-24T00:36:41.074257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080483+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:36:41.080483+0000","last_peered":"2024-03-24T00:36:41.080483+0000","last_clean":"2024-03-24T00:36:41.080483+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:36:41.080483+0000","last_undegraded":"2024-03-24T00:36:41.080483+0000","last_fullsized":"2024-03-24T00:36:41.080483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075653+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:36:41.075653+0000","last_peered":"2024-03-24T00:36:41.075653+0000","last_clean":"2024-03-24T00:36:41.075653+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:36:41.075653+0000","last_undegraded":"2024-03-24T00:36:41.075653+0000","last_fullsized":"2024-03-24T00:36:41.075653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056352+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:36:41.056352+0000","last_peered":"2024-03-24T00:36:41.056352+0000","last_clean":"2024-03-24T00:36:41.056352+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:36:41.056352+0000","last_undegraded":"2024-03-24T00:36:41.056352+0000","last_fullsized":"2024-03-24T00:36:41.056352+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061383+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:36:41.061383+0000","last_peered":"2024-03-24T00:36:41.061383+0000","last_clean":"2024-03-24T00:36:41.061383+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:36:41.061383+0000","last_undegraded":"2024-03-24T00:36:41.061383+0000","last_fullsized":"2024-03-24T00:36:41.061383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063992+0000","last_change":"2024-03-24T00:35:42.437175+0000","last_active":"2024-03-24T00:36:41.063992+0000","last_peered":"2024-03-24T00:36:41.063992+0000","last_clean":"2024-03-24T00:36:41.063992+0000","last_became_active":"2024-03-24T00:35:42.436625+0000","last_became_peered":"2024-03-24T00:35:42.436625+0000","last_unstale":"2024-03-24T00:36:41.063992+0000","last_undegraded":"2024-03-24T00:36:41.063992+0000","last_fullsized":"2024-03-24T00:36:41.063992+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:29:26.922046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064788+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:36:41.064788+0000","last_peered":"2024-03-24T00:36:41.064788+0000","last_clean":"2024-03-24T00:36:41.064788+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:36:41.064788+0000","last_undegraded":"2024-03-24T00:36:41.064788+0000","last_fullsized":"2024-03-24T00:36:41.064788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063770+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:36:41.063770+0000","last_peered":"2024-03-24T00:36:41.063770+0000","last_clean":"2024-03-24T00:36:41.063770+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:36:41.063770+0000","last_undegraded":"2024-03-24T00:36:41.063770+0000","last_fullsized":"2024-03-24T00:36:41.063770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064615+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:36:41.064615+0000","last_peered":"2024-03-24T00:36:41.064615+0000","last_clean":"2024-03-24T00:36:41.064615+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:36:41.064615+0000","last_undegraded":"2024-03-24T00:36:41.064615+0000","last_fullsized":"2024-03-24T00:36:41.064615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.081747+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:36:41.081747+0000","last_peered":"2024-03-24T00:36:41.081747+0000","last_clean":"2024-03-24T00:36:41.081747+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:36:41.081747+0000","last_undegraded":"2024-03-24T00:36:41.081747+0000","last_fullsized":"2024-03-24T00:36:41.081747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056973+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:36:41.056973+0000","last_peered":"2024-03-24T00:36:41.056973+0000","last_clean":"2024-03-24T00:36:41.056973+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:36:41.056973+0000","last_undegraded":"2024-03-24T00:36:41.056973+0000","last_fullsized":"2024-03-24T00:36:41.056973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62943,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":172,"num_read_kb":149,"num_write":84,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":188829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318304,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579940,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1118740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037807,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1056983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462969,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":528061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":528095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":466254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1056583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:36:59.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:37:00.110 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:00.551 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:00 smithi123 ceph-mon[31487]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:00.551 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:00 smithi123 ceph-mon[31487]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:00 smithi204 ceph-mon[36345]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:00 smithi204 ceph-mon[36345]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:00 smithi177 ceph-mon[36034]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:00 smithi177 ceph-mon[36034]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:02.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 -- 172.21.15.123:0/3972433351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee24074ce0 msgr2=0x7fee24112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 --2- 172.21.15.123:0/3972433351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee24074ce0 0x7fee24112e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009990 tx=0x7fee0c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 -- 172.21.15.123:0/3972433351 shutdown_connections 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 --2- 172.21.15.123:0/3972433351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24113360 0x7fee24115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 --2- 172.21.15.123:0/3972433351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240743c0 0x7fee240747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 --2- 172.21.15.123:0/3972433351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee24074ce0 0x7fee24112e20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.401+0000 7fee29e00700 1 -- 172.21.15.123:0/3972433351 >> 172.21.15.123:0/3972433351 conn(0x7fee240792f0 msgr2=0x7fee24079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.403+0000 7fee29e00700 1 -- 172.21.15.123:0/3972433351 shutdown_connections 2024-03-24T00:37:02.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.403+0000 7fee29e00700 1 -- 172.21.15.123:0/3972433351 wait complete. 2024-03-24T00:37:02.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 Processor -- start 2024-03-24T00:37:02.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 -- start start 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24074ce0 0x7fee2419f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 0x7fee241a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24118bb0 con 0x7fee24113360 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24118a30 con 0x7fee240743c0 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee29e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24118d30 con 0x7fee24074ce0 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.404+0000 7fee22a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:42380/0 (socket says 172.21.15.123:42380) 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 -- 172.21.15.123:0/3914925179 learned_addr learned my addr 172.21.15.123:0/3914925179 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee1a000700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24074ce0 0x7fee2419f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee23400700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 0x7fee241a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 -- 172.21.15.123:0/3914925179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24074ce0 msgr2=0x7fee2419f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24074ce0 0x7fee2419f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 -- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 msgr2=0x7fee241a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 0x7fee241a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 -- 172.21.15.123:0/3914925179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee241a3fb0 con 0x7fee240743c0 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee23400700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 0x7fee241a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee22a00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fee1400eb30 tx=0x7fee1400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1400cc90 con 0x7fee240743c0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee14004500 con 0x7fee240743c0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.405+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14010430 con 0x7fee240743c0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.406+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee241a42a0 con 0x7fee240743c0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.406+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee241abb20 con 0x7fee240743c0 2024-03-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.407+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee24068e50 con 0x7fee240743c0 2024-03-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.408+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee14003f70 con 0x7fee240743c0 2024-03-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.410+0000 7fee28a00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 0x7fee080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.410+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fee140cee10 con 0x7fee240743c0 2024-03-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.410+0000 7fee1a000700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 0x7fee080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.411+0000 7fee1a000700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 0x7fee080a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fee24112720 tx=0x7fee0c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.414+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee140d2030 con 0x7fee240743c0 2024-03-24T00:37:02.513 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:02 smithi177 ceph-mon[36034]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:02 smithi204 ceph-mon[36345]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:02 smithi123 ceph-mon[31487]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:02.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.635+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee2410f510 con 0x7fee080a2890 2024-03-24T00:37:02.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.656+0000 7fee28a00700 1 -- 172.21.15.123:0/3914925179 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258751 (secure 0 0 0) 0x7fee0c0652d0 con 0x7fee080a2890 2024-03-24T00:37:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.661+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 msgr2=0x7fee080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.661+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 0x7fee080a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fee24112720 tx=0x7fee0c00b260 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.661+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 msgr2=0x7fee2419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.661+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fee1400eb30 tx=0x7fee1400ee40 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 shutdown_connections 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee080a2890 0x7fee080a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee24113360 0x7fee241a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee240743c0 0x7fee2419efa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 --2- 172.21.15.123:0/3914925179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee24074ce0 0x7fee2419f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 >> 172.21.15.123:0/3914925179 conn(0x7fee240792f0 msgr2=0x7fee2410ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 shutdown_connections 2024-03-24T00:37:02.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:02.662+0000 7fee29e00700 1 -- 172.21.15.123:0/3914925179 wait complete. 2024-03-24T00:37:02.665 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:37:03.516 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-03-24T00:37:01.103398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653545,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":308,"num_read_kb":263,"num_write":267,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":709,"ondisk_log_size":709,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354104,"kb_used_data":13272,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587720,"statfs":{"total":1151940427776,"available":1151577825280,"internally_reserved":0,"allocated":13590528,"data_stored":7551627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005283"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082727+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:36:41.082727+0000","last_peered":"2024-03-24T00:36:41.082727+0000","last_clean":"2024-03-24T00:36:41.082727+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:36:41.082727+0000","last_undegraded":"2024-03-24T00:36:41.082727+0000","last_fullsized":"2024-03-24T00:36:41.082727+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082980+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:36:41.082980+0000","last_peered":"2024-03-24T00:36:41.082980+0000","last_clean":"2024-03-24T00:36:41.082980+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:36:41.082980+0000","last_undegraded":"2024-03-24T00:36:41.082980+0000","last_fullsized":"2024-03-24T00:36:41.082980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082446+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:36:41.082446+0000","last_peered":"2024-03-24T00:36:41.082446+0000","last_clean":"2024-03-24T00:36:41.082446+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:36:41.082446+0000","last_undegraded":"2024-03-24T00:36:41.082446+0000","last_fullsized":"2024-03-24T00:36:41.082446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.058888+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:36:41.058888+0000","last_peered":"2024-03-24T00:36:41.058888+0000","last_clean":"2024-03-24T00:36:41.058888+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:36:41.058888+0000","last_undegraded":"2024-03-24T00:36:41.058888+0000","last_fullsized":"2024-03-24T00:36:41.058888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.068426+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:36:41.068426+0000","last_peered":"2024-03-24T00:36:41.068426+0000","last_clean":"2024-03-24T00:36:41.068426+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:36:41.068426+0000","last_undegraded":"2024-03-24T00:36:41.068426+0000","last_fullsized":"2024-03-24T00:36:41.068426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073848+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:36:41.073848+0000","last_peered":"2024-03-24T00:36:41.073848+0000","last_clean":"2024-03-24T00:36:41.073848+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:36:41.073848+0000","last_undegraded":"2024-03-24T00:36:41.073848+0000","last_fullsized":"2024-03-24T00:36:41.073848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074753+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:36:41.074753+0000","last_peered":"2024-03-24T00:36:41.074753+0000","last_clean":"2024-03-24T00:36:41.074753+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:36:41.074753+0000","last_undegraded":"2024-03-24T00:36:41.074753+0000","last_fullsized":"2024-03-24T00:36:41.074753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065233+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:36:41.065233+0000","last_peered":"2024-03-24T00:36:41.065233+0000","last_clean":"2024-03-24T00:36:41.065233+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:36:41.065233+0000","last_undegraded":"2024-03-24T00:36:41.065233+0000","last_fullsized":"2024-03-24T00:36:41.065233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063340+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:36:41.063340+0000","last_peered":"2024-03-24T00:36:41.063340+0000","last_clean":"2024-03-24T00:36:41.063340+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:36:41.063340+0000","last_undegraded":"2024-03-24T00:36:41.063340+0000","last_fullsized":"2024-03-24T00:36:41.063340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078500+0000","last_change":"2024-03-24T00:35:42.436405+0000","last_active":"2024-03-24T00:36:41.078500+0000","last_peered":"2024-03-24T00:36:41.078500+0000","last_clean":"2024-03-24T00:36:41.078500+0000","last_became_active":"2024-03-24T00:35:42.436022+0000","last_became_peered":"2024-03-24T00:35:42.436022+0000","last_unstale":"2024-03-24T00:36:41.078500+0000","last_undegraded":"2024-03-24T00:36:41.078500+0000","last_fullsized":"2024-03-24T00:36:41.078500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:59.022152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063157+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:36:41.063157+0000","last_peered":"2024-03-24T00:36:41.063157+0000","last_clean":"2024-03-24T00:36:41.063157+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:36:41.063157+0000","last_undegraded":"2024-03-24T00:36:41.063157+0000","last_fullsized":"2024-03-24T00:36:41.063157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.046590+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:36:41.046590+0000","last_peered":"2024-03-24T00:36:41.046590+0000","last_clean":"2024-03-24T00:36:41.046590+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:36:41.046590+0000","last_undegraded":"2024-03-24T00:36:41.046590+0000","last_fullsized":"2024-03-24T00:36:41.046590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.077839+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:36:41.077839+0000","last_peered":"2024-03-24T00:36:41.077839+0000","last_clean":"2024-03-24T00:36:41.077839+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:36:41.077839+0000","last_undegraded":"2024-03-24T00:36:41.077839+0000","last_fullsized":"2024-03-24T00:36:41.077839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.084192+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:36:41.084192+0000","last_peered":"2024-03-24T00:36:41.084192+0000","last_clean":"2024-03-24T00:36:41.084192+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:36:41.084192+0000","last_undegraded":"2024-03-24T00:36:41.084192+0000","last_fullsized":"2024-03-24T00:36:41.084192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083963+0000","last_change":"2024-03-24T00:35:42.433339+0000","last_active":"2024-03-24T00:36:41.083963+0000","last_peered":"2024-03-24T00:36:41.083963+0000","last_clean":"2024-03-24T00:36:41.083963+0000","last_became_active":"2024-03-24T00:35:42.432819+0000","last_became_peered":"2024-03-24T00:35:42.432819+0000","last_unstale":"2024-03-24T00:36:41.083963+0000","last_undegraded":"2024-03-24T00:36:41.083963+0000","last_fullsized":"2024-03-24T00:36:41.083963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:09.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082904+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:36:41.082904+0000","last_peered":"2024-03-24T00:36:41.082904+0000","last_clean":"2024-03-24T00:36:41.082904+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:36:41.082904+0000","last_undegraded":"2024-03-24T00:36:41.082904+0000","last_fullsized":"2024-03-24T00:36:41.082904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082789+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:36:41.082789+0000","last_peered":"2024-03-24T00:36:41.082789+0000","last_clean":"2024-03-24T00:36:41.082789+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:36:41.082789+0000","last_undegraded":"2024-03-24T00:36:41.082789+0000","last_fullsized":"2024-03-24T00:36:41.082789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.079303+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:36:41.079303+0000","last_peered":"2024-03-24T00:36:41.079303+0000","last_clean":"2024-03-24T00:36:41.079303+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:36:41.079303+0000","last_undegraded":"2024-03-24T00:36:41.079303+0000","last_fullsized":"2024-03-24T00:36:41.079303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056499+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:36:41.056499+0000","last_peered":"2024-03-24T00:36:41.056499+0000","last_clean":"2024-03-24T00:36:41.056499+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:36:41.056499+0000","last_undegraded":"2024-03-24T00:36:41.056499+0000","last_fullsized":"2024-03-24T00:36:41.056499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435460+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:42.435460+0000","last_peered":"2024-03-24T00:35:42.435460+0000","last_clean":"2024-03-24T00:35:42.435460+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:42.435460+0000","last_undegraded":"2024-03-24T00:35:42.435460+0000","last_fullsized":"2024-03-24T00:35:42.435460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062812+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:36:41.062812+0000","last_peered":"2024-03-24T00:36:41.062812+0000","last_clean":"2024-03-24T00:36:41.062812+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:36:41.062812+0000","last_undegraded":"2024-03-24T00:36:41.062812+0000","last_fullsized":"2024-03-24T00:36:41.062812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.433298+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:42.433298+0000","last_peered":"2024-03-24T00:35:42.433298+0000","last_clean":"2024-03-24T00:35:42.433298+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:42.433298+0000","last_undegraded":"2024-03-24T00:35:42.433298+0000","last_fullsized":"2024-03-24T00:35:42.433298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080791+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:36:41.080791+0000","last_peered":"2024-03-24T00:36:41.080791+0000","last_clean":"2024-03-24T00:36:41.080791+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:36:41.080791+0000","last_undegraded":"2024-03-24T00:36:41.080791+0000","last_fullsized":"2024-03-24T00:36:41.080791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435536+0000","last_change":"2024-03-24T00:35:42.435006+0000","last_active":"2024-03-24T00:35:42.435536+0000","last_peered":"2024-03-24T00:35:42.435536+0000","last_clean":"2024-03-24T00:35:42.435536+0000","last_became_active":"2024-03-24T00:35:42.429558+0000","last_became_peered":"2024-03-24T00:35:42.429558+0000","last_unstale":"2024-03-24T00:35:42.435536+0000","last_undegraded":"2024-03-24T00:35:42.435536+0000","last_fullsized":"2024-03-24T00:35:42.435536+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:09.212462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061003+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:36:41.061003+0000","last_peered":"2024-03-24T00:36:41.061003+0000","last_clean":"2024-03-24T00:36:41.061003+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:36:41.061003+0000","last_undegraded":"2024-03-24T00:36:41.061003+0000","last_fullsized":"2024-03-24T00:36:41.061003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431452+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:42.431452+0000","last_peered":"2024-03-24T00:35:42.431452+0000","last_clean":"2024-03-24T00:35:42.431452+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:42.431452+0000","last_undegraded":"2024-03-24T00:35:42.431452+0000","last_fullsized":"2024-03-24T00:35:42.431452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431007+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:42.431007+0000","last_peered":"2024-03-24T00:35:42.431007+0000","last_clean":"2024-03-24T00:35:42.431007+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:42.431007+0000","last_undegraded":"2024-03-24T00:35:42.431007+0000","last_fullsized":"2024-03-24T00:35:42.431007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074671+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:36:41.074671+0000","last_peered":"2024-03-24T00:36:41.074671+0000","last_clean":"2024-03-24T00:36:41.074671+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:36:41.074671+0000","last_undegraded":"2024-03-24T00:36:41.074671+0000","last_fullsized":"2024-03-24T00:36:41.074671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422152+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:42.422152+0000","last_peered":"2024-03-24T00:35:42.422152+0000","last_clean":"2024-03-24T00:35:42.422152+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:42.422152+0000","last_undegraded":"2024-03-24T00:35:42.422152+0000","last_fullsized":"2024-03-24T00:35:42.422152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061844+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:36:41.061844+0000","last_peered":"2024-03-24T00:36:41.061844+0000","last_clean":"2024-03-24T00:36:41.061844+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:36:41.061844+0000","last_undegraded":"2024-03-24T00:36:41.061844+0000","last_fullsized":"2024-03-24T00:36:41.061844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422989+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:42.422989+0000","last_peered":"2024-03-24T00:35:42.422989+0000","last_clean":"2024-03-24T00:35:42.422989+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:42.422989+0000","last_undegraded":"2024-03-24T00:35:42.422989+0000","last_fullsized":"2024-03-24T00:35:42.422989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071616+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:36:41.071616+0000","last_peered":"2024-03-24T00:36:41.071616+0000","last_clean":"2024-03-24T00:36:41.071616+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:36:41.071616+0000","last_undegraded":"2024-03-24T00:36:41.071616+0000","last_fullsized":"2024-03-24T00:36:41.071616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430389+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:42.430389+0000","last_peered":"2024-03-24T00:35:42.430389+0000","last_clean":"2024-03-24T00:35:42.430389+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:42.430389+0000","last_undegraded":"2024-03-24T00:35:42.430389+0000","last_fullsized":"2024-03-24T00:35:42.430389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073853+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:36:41.073853+0000","last_peered":"2024-03-24T00:36:41.073853+0000","last_clean":"2024-03-24T00:36:41.073853+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:36:41.073853+0000","last_undegraded":"2024-03-24T00:36:41.073853+0000","last_fullsized":"2024-03-24T00:36:41.073853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422441+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:42.422441+0000","last_peered":"2024-03-24T00:35:42.422441+0000","last_clean":"2024-03-24T00:35:42.422441+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:42.422441+0000","last_undegraded":"2024-03-24T00:35:42.422441+0000","last_fullsized":"2024-03-24T00:35:42.422441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078541+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:36:41.078541+0000","last_peered":"2024-03-24T00:36:41.078541+0000","last_clean":"2024-03-24T00:36:41.078541+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:36:41.078541+0000","last_undegraded":"2024-03-24T00:36:41.078541+0000","last_fullsized":"2024-03-24T00:36:41.078541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423580+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:42.423580+0000","last_peered":"2024-03-24T00:35:42.423580+0000","last_clean":"2024-03-24T00:35:42.423580+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:42.423580+0000","last_undegraded":"2024-03-24T00:35:42.423580+0000","last_fullsized":"2024-03-24T00:35:42.423580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083152+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:36:41.083152+0000","last_peered":"2024-03-24T00:36:41.083152+0000","last_clean":"2024-03-24T00:36:41.083152+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:36:41.083152+0000","last_undegraded":"2024-03-24T00:36:41.083152+0000","last_fullsized":"2024-03-24T00:36:41.083152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431398+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:42.431398+0000","last_peered":"2024-03-24T00:35:42.431398+0000","last_clean":"2024-03-24T00:35:42.431398+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:42.431398+0000","last_undegraded":"2024-03-24T00:35:42.431398+0000","last_fullsized":"2024-03-24T00:35:42.431398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057322+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:36:41.057322+0000","last_peered":"2024-03-24T00:36:41.057322+0000","last_clean":"2024-03-24T00:36:41.057322+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:36:41.057322+0000","last_undegraded":"2024-03-24T00:36:41.057322+0000","last_fullsized":"2024-03-24T00:36:41.057322+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":225,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:59.207930+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:36:59.207930+0000","last_peered":"2024-03-24T00:36:59.207930+0000","last_clean":"2024-03-24T00:36:59.207930+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:36:59.207930+0000","last_undegraded":"2024-03-24T00:36:59.207930+0000","last_fullsized":"2024-03-24T00:36:59.207930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62145,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":149,"num_write":29,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062432+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:36:41.062432+0000","last_peered":"2024-03-24T00:36:41.062432+0000","last_clean":"2024-03-24T00:36:41.062432+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:36:41.062432+0000","last_undegraded":"2024-03-24T00:36:41.062432+0000","last_fullsized":"2024-03-24T00:36:41.062432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424521+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:42.424521+0000","last_peered":"2024-03-24T00:35:42.424521+0000","last_clean":"2024-03-24T00:35:42.424521+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:42.424521+0000","last_undegraded":"2024-03-24T00:35:42.424521+0000","last_fullsized":"2024-03-24T00:35:42.424521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067324+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:36:41.067324+0000","last_peered":"2024-03-24T00:36:41.067324+0000","last_clean":"2024-03-24T00:36:41.067324+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:36:41.067324+0000","last_undegraded":"2024-03-24T00:36:41.067324+0000","last_fullsized":"2024-03-24T00:36:41.067324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430861+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:42.430861+0000","last_peered":"2024-03-24T00:35:42.430861+0000","last_clean":"2024-03-24T00:35:42.430861+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:42.430861+0000","last_undegraded":"2024-03-24T00:35:42.430861+0000","last_fullsized":"2024-03-24T00:35:42.430861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078222+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:36:41.078222+0000","last_peered":"2024-03-24T00:36:41.078222+0000","last_clean":"2024-03-24T00:36:41.078222+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:36:41.078222+0000","last_undegraded":"2024-03-24T00:36:41.078222+0000","last_fullsized":"2024-03-24T00:36:41.078222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425512+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:42.425512+0000","last_peered":"2024-03-24T00:35:42.425512+0000","last_clean":"2024-03-24T00:35:42.425512+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:42.425512+0000","last_undegraded":"2024-03-24T00:35:42.425512+0000","last_fullsized":"2024-03-24T00:35:42.425512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067274+0000","last_change":"2024-03-24T00:35:42.436171+0000","last_active":"2024-03-24T00:36:41.067274+0000","last_peered":"2024-03-24T00:36:41.067274+0000","last_clean":"2024-03-24T00:36:41.067274+0000","last_became_active":"2024-03-24T00:35:42.435757+0000","last_became_peered":"2024-03-24T00:35:42.435757+0000","last_unstale":"2024-03-24T00:36:41.067274+0000","last_undegraded":"2024-03-24T00:36:41.067274+0000","last_fullsized":"2024-03-24T00:36:41.067274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:08.194563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430606+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:42.430606+0000","last_peered":"2024-03-24T00:35:42.430606+0000","last_clean":"2024-03-24T00:35:42.430606+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:42.430606+0000","last_undegraded":"2024-03-24T00:35:42.430606+0000","last_fullsized":"2024-03-24T00:35:42.430606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059832+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:36:41.059832+0000","last_peered":"2024-03-24T00:36:41.059832+0000","last_clean":"2024-03-24T00:36:41.059832+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:36:41.059832+0000","last_undegraded":"2024-03-24T00:36:41.059832+0000","last_fullsized":"2024-03-24T00:36:41.059832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.438512+0000","last_change":"2024-03-24T00:35:42.438099+0000","last_active":"2024-03-24T00:35:42.438512+0000","last_peered":"2024-03-24T00:35:42.438512+0000","last_clean":"2024-03-24T00:35:42.438512+0000","last_became_active":"2024-03-24T00:35:42.432549+0000","last_became_peered":"2024-03-24T00:35:42.432549+0000","last_unstale":"2024-03-24T00:35:42.438512+0000","last_undegraded":"2024-03-24T00:35:42.438512+0000","last_fullsized":"2024-03-24T00:35:42.438512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:33.579265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.053232+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:36:41.053232+0000","last_peered":"2024-03-24T00:36:41.053232+0000","last_clean":"2024-03-24T00:36:41.053232+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:36:41.053232+0000","last_undegraded":"2024-03-24T00:36:41.053232+0000","last_fullsized":"2024-03-24T00:36:41.053232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.434686+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:42.434686+0000","last_peered":"2024-03-24T00:35:42.434686+0000","last_clean":"2024-03-24T00:35:42.434686+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:42.434686+0000","last_undegraded":"2024-03-24T00:35:42.434686+0000","last_fullsized":"2024-03-24T00:35:42.434686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.049175+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:36:41.049175+0000","last_peered":"2024-03-24T00:36:41.049175+0000","last_clean":"2024-03-24T00:36:41.049175+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:36:41.049175+0000","last_undegraded":"2024-03-24T00:36:41.049175+0000","last_fullsized":"2024-03-24T00:36:41.049175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423519+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:42.423519+0000","last_peered":"2024-03-24T00:35:42.423519+0000","last_clean":"2024-03-24T00:35:42.423519+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:42.423519+0000","last_undegraded":"2024-03-24T00:35:42.423519+0000","last_fullsized":"2024-03-24T00:35:42.423519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082180+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:36:41.082180+0000","last_peered":"2024-03-24T00:36:41.082180+0000","last_clean":"2024-03-24T00:36:41.082180+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:36:41.082180+0000","last_undegraded":"2024-03-24T00:36:41.082180+0000","last_fullsized":"2024-03-24T00:36:41.082180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.051429+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:36:41.051429+0000","last_peered":"2024-03-24T00:36:41.051429+0000","last_clean":"2024-03-24T00:36:41.051429+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:36:41.051429+0000","last_undegraded":"2024-03-24T00:36:41.051429+0000","last_fullsized":"2024-03-24T00:36:41.051429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422884+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:42.422884+0000","last_peered":"2024-03-24T00:35:42.422884+0000","last_clean":"2024-03-24T00:35:42.422884+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:42.422884+0000","last_undegraded":"2024-03-24T00:35:42.422884+0000","last_fullsized":"2024-03-24T00:35:42.422884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063946+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:36:41.063946+0000","last_peered":"2024-03-24T00:36:41.063946+0000","last_clean":"2024-03-24T00:36:41.063946+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:36:41.063946+0000","last_undegraded":"2024-03-24T00:36:41.063946+0000","last_fullsized":"2024-03-24T00:36:41.063946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425608+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:42.425608+0000","last_peered":"2024-03-24T00:35:42.425608+0000","last_clean":"2024-03-24T00:35:42.425608+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:42.425608+0000","last_undegraded":"2024-03-24T00:35:42.425608+0000","last_fullsized":"2024-03-24T00:35:42.425608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423432+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:42.423432+0000","last_peered":"2024-03-24T00:35:42.423432+0000","last_clean":"2024-03-24T00:35:42.423432+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:42.423432+0000","last_undegraded":"2024-03-24T00:35:42.423432+0000","last_fullsized":"2024-03-24T00:35:42.423432+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059155+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:36:41.059155+0000","last_peered":"2024-03-24T00:36:41.059155+0000","last_clean":"2024-03-24T00:36:41.059155+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:36:41.059155+0000","last_undegraded":"2024-03-24T00:36:41.059155+0000","last_fullsized":"2024-03-24T00:36:41.059155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423281+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:42.423281+0000","last_peered":"2024-03-24T00:35:42.423281+0000","last_clean":"2024-03-24T00:35:42.423281+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:42.423281+0000","last_undegraded":"2024-03-24T00:35:42.423281+0000","last_fullsized":"2024-03-24T00:35:42.423281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431457+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:42.431457+0000","last_peered":"2024-03-24T00:35:42.431457+0000","last_clean":"2024-03-24T00:35:42.431457+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:42.431457+0000","last_undegraded":"2024-03-24T00:35:42.431457+0000","last_fullsized":"2024-03-24T00:35:42.431457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074539+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:36:41.074539+0000","last_peered":"2024-03-24T00:36:41.074539+0000","last_clean":"2024-03-24T00:36:41.074539+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:36:41.074539+0000","last_undegraded":"2024-03-24T00:36:41.074539+0000","last_fullsized":"2024-03-24T00:36:41.074539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:46.358125+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:35:46.358125+0000","last_peered":"2024-03-24T00:35:46.358125+0000","last_clean":"2024-03-24T00:35:46.358125+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:35:46.358125+0000","last_undegraded":"2024-03-24T00:35:46.358125+0000","last_fullsized":"2024-03-24T00:35:46.358125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065991+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:36:41.065991+0000","last_peered":"2024-03-24T00:36:41.065991+0000","last_clean":"2024-03-24T00:36:41.065991+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:36:41.065991+0000","last_undegraded":"2024-03-24T00:36:41.065991+0000","last_fullsized":"2024-03-24T00:36:41.065991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425899+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:42.425899+0000","last_peered":"2024-03-24T00:35:42.425899+0000","last_clean":"2024-03-24T00:35:42.425899+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:42.425899+0000","last_undegraded":"2024-03-24T00:35:42.425899+0000","last_fullsized":"2024-03-24T00:35:42.425899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075280+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:36:41.075280+0000","last_peered":"2024-03-24T00:36:41.075280+0000","last_clean":"2024-03-24T00:36:41.075280+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:36:41.075280+0000","last_undegraded":"2024-03-24T00:36:41.075280+0000","last_fullsized":"2024-03-24T00:36:41.075280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.862632+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:42.862632+0000","last_peered":"2024-03-24T00:35:42.862632+0000","last_clean":"2024-03-24T00:35:42.862632+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:42.862632+0000","last_undegraded":"2024-03-24T00:35:42.862632+0000","last_fullsized":"2024-03-24T00:35:42.862632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.024641+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:36:41.024641+0000","last_peered":"2024-03-24T00:36:41.024641+0000","last_clean":"2024-03-24T00:36:41.024641+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:36:41.024641+0000","last_undegraded":"2024-03-24T00:36:41.024641+0000","last_fullsized":"2024-03-24T00:36:41.024641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.428625+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:42.428625+0000","last_peered":"2024-03-24T00:35:42.428625+0000","last_clean":"2024-03-24T00:35:42.428625+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:42.428625+0000","last_undegraded":"2024-03-24T00:35:42.428625+0000","last_fullsized":"2024-03-24T00:35:42.428625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082612+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:36:41.082612+0000","last_peered":"2024-03-24T00:36:41.082612+0000","last_clean":"2024-03-24T00:36:41.082612+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:36:41.082612+0000","last_undegraded":"2024-03-24T00:36:41.082612+0000","last_fullsized":"2024-03-24T00:36:41.082612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424512+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:42.424512+0000","last_peered":"2024-03-24T00:35:42.424512+0000","last_clean":"2024-03-24T00:35:42.424512+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:42.424512+0000","last_undegraded":"2024-03-24T00:35:42.424512+0000","last_fullsized":"2024-03-24T00:35:42.424512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080523+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:36:41.080523+0000","last_peered":"2024-03-24T00:36:41.080523+0000","last_clean":"2024-03-24T00:36:41.080523+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:36:41.080523+0000","last_undegraded":"2024-03-24T00:36:41.080523+0000","last_fullsized":"2024-03-24T00:36:41.080523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430730+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:42.430730+0000","last_peered":"2024-03-24T00:35:42.430730+0000","last_clean":"2024-03-24T00:35:42.430730+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:42.430730+0000","last_undegraded":"2024-03-24T00:35:42.430730+0000","last_fullsized":"2024-03-24T00:35:42.430730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071383+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:36:41.071383+0000","last_peered":"2024-03-24T00:36:41.071383+0000","last_clean":"2024-03-24T00:36:41.071383+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:36:41.071383+0000","last_undegraded":"2024-03-24T00:36:41.071383+0000","last_fullsized":"2024-03-24T00:36:41.071383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.432672+0000","last_change":"2024-03-24T00:35:42.432365+0000","last_active":"2024-03-24T00:35:42.432672+0000","last_peered":"2024-03-24T00:35:42.432672+0000","last_clean":"2024-03-24T00:35:42.432672+0000","last_became_active":"2024-03-24T00:35:42.432113+0000","last_became_peered":"2024-03-24T00:35:42.432113+0000","last_unstale":"2024-03-24T00:35:42.432672+0000","last_undegraded":"2024-03-24T00:35:42.432672+0000","last_fullsized":"2024-03-24T00:35:42.432672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:32.467534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065060+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:36:41.065060+0000","last_peered":"2024-03-24T00:36:41.065060+0000","last_clean":"2024-03-24T00:36:41.065060+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:36:41.065060+0000","last_undegraded":"2024-03-24T00:36:41.065060+0000","last_fullsized":"2024-03-24T00:36:41.065060+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431200+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:42.431200+0000","last_peered":"2024-03-24T00:35:42.431200+0000","last_clean":"2024-03-24T00:35:42.431200+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:42.431200+0000","last_undegraded":"2024-03-24T00:35:42.431200+0000","last_fullsized":"2024-03-24T00:35:42.431200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063211+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:36:41.063211+0000","last_peered":"2024-03-24T00:36:41.063211+0000","last_clean":"2024-03-24T00:36:41.063211+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:36:41.063211+0000","last_undegraded":"2024-03-24T00:36:41.063211+0000","last_fullsized":"2024-03-24T00:36:41.063211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.866804+0000","last_change":"2024-03-24T00:35:42.866804+0000","last_active":"2024-03-24T00:35:42.866804+0000","last_peered":"2024-03-24T00:35:42.866804+0000","last_clean":"2024-03-24T00:35:42.866804+0000","last_became_active":"2024-03-24T00:35:42.863933+0000","last_became_peered":"2024-03-24T00:35:42.863933+0000","last_unstale":"2024-03-24T00:35:42.866804+0000","last_undegraded":"2024-03-24T00:35:42.866804+0000","last_fullsized":"2024-03-24T00:35:42.866804+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:55.475527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062049+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:36:41.062049+0000","last_peered":"2024-03-24T00:36:41.062049+0000","last_clean":"2024-03-24T00:36:41.062049+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:36:41.062049+0000","last_undegraded":"2024-03-24T00:36:41.062049+0000","last_fullsized":"2024-03-24T00:36:41.062049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.052456+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:36:41.052456+0000","last_peered":"2024-03-24T00:36:41.052456+0000","last_clean":"2024-03-24T00:36:41.052456+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:36:41.052456+0000","last_undegraded":"2024-03-24T00:36:41.052456+0000","last_fullsized":"2024-03-24T00:36:41.052456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074257+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:36:41.074257+0000","last_peered":"2024-03-24T00:36:41.074257+0000","last_clean":"2024-03-24T00:36:41.074257+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:36:41.074257+0000","last_undegraded":"2024-03-24T00:36:41.074257+0000","last_fullsized":"2024-03-24T00:36:41.074257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080483+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:36:41.080483+0000","last_peered":"2024-03-24T00:36:41.080483+0000","last_clean":"2024-03-24T00:36:41.080483+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:36:41.080483+0000","last_undegraded":"2024-03-24T00:36:41.080483+0000","last_fullsized":"2024-03-24T00:36:41.080483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075653+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:36:41.075653+0000","last_peered":"2024-03-24T00:36:41.075653+0000","last_clean":"2024-03-24T00:36:41.075653+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:36:41.075653+0000","last_undegraded":"2024-03-24T00:36:41.075653+0000","last_fullsized":"2024-03-24T00:36:41.075653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056352+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:36:41.056352+0000","last_peered":"2024-03-24T00:36:41.056352+0000","last_clean":"2024-03-24T00:36:41.056352+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:36:41.056352+0000","last_undegraded":"2024-03-24T00:36:41.056352+0000","last_fullsized":"2024-03-24T00:36:41.056352+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061383+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:36:41.061383+0000","last_peered":"2024-03-24T00:36:41.061383+0000","last_clean":"2024-03-24T00:36:41.061383+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:36:41.061383+0000","last_undegraded":"2024-03-24T00:36:41.061383+0000","last_fullsized":"2024-03-24T00:36:41.061383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063992+0000","last_change":"2024-03-24T00:35:42.437175+0000","last_active":"2024-03-24T00:36:41.063992+0000","last_peered":"2024-03-24T00:36:41.063992+0000","last_clean":"2024-03-24T00:36:41.063992+0000","last_became_active":"2024-03-24T00:35:42.436625+0000","last_became_peered":"2024-03-24T00:35:42.436625+0000","last_unstale":"2024-03-24T00:36:41.063992+0000","last_undegraded":"2024-03-24T00:36:41.063992+0000","last_fullsized":"2024-03-24T00:36:41.063992+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:29:26.922046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064788+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:36:41.064788+0000","last_peered":"2024-03-24T00:36:41.064788+0000","last_clean":"2024-03-24T00:36:41.064788+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:36:41.064788+0000","last_undegraded":"2024-03-24T00:36:41.064788+0000","last_fullsized":"2024-03-24T00:36:41.064788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063770+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:36:41.063770+0000","last_peered":"2024-03-24T00:36:41.063770+0000","last_clean":"2024-03-24T00:36:41.063770+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:36:41.063770+0000","last_undegraded":"2024-03-24T00:36:41.063770+0000","last_fullsized":"2024-03-24T00:36:41.063770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064615+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:36:41.064615+0000","last_peered":"2024-03-24T00:36:41.064615+0000","last_clean":"2024-03-24T00:36:41.064615+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:36:41.064615+0000","last_undegraded":"2024-03-24T00:36:41.064615+0000","last_fullsized":"2024-03-24T00:36:41.064615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.081747+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:36:41.081747+0000","last_peered":"2024-03-24T00:36:41.081747+0000","last_clean":"2024-03-24T00:36:41.081747+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:36:41.081747+0000","last_undegraded":"2024-03-24T00:36:41.081747+0000","last_fullsized":"2024-03-24T00:36:41.081747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056973+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:36:41.056973+0000","last_peered":"2024-03-24T00:36:41.056973+0000","last_clean":"2024-03-24T00:36:41.056973+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:36:41.056973+0000","last_undegraded":"2024-03-24T00:36:41.056973+0000","last_fullsized":"2024-03-24T00:36:41.056973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62943,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":182,"num_read_kb":154,"num_write":84,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":188829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318305,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579941,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1118740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1056983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":528061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":528095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":466254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1056583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:03.520 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T00:37:03.520 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T00:37:03.520 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T00:37:03.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T00:37:04.009 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:04.426 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:04 smithi123 ceph-mon[31487]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:04.426 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:04 smithi123 ceph-mon[31487]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:04 smithi204 ceph-mon[36345]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:04 smithi204 ceph-mon[36345]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:04 smithi177 ceph-mon[36034]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:04 smithi177 ceph-mon[36034]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 -- 172.21.15.123:0/3805566806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 msgr2=0x7f41cc10e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 --2- 172.21.15.123:0/3805566806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc10e370 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009990 tx=0x7f41b4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 -- 172.21.15.123:0/3805566806 shutdown_connections 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 --2- 172.21.15.123:0/3805566806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 0x7f41cc110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 --2- 172.21.15.123:0/3805566806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 0x7f41cc0749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 --2- 172.21.15.123:0/3805566806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc10e370 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 -- 172.21.15.123:0/3805566806 >> 172.21.15.123:0/3805566806 conn(0x7f41cc0fdb60 msgr2=0x7f41cc0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 -- 172.21.15.123:0/3805566806 shutdown_connections 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.086+0000 7f41d0c00700 1 -- 172.21.15.123:0/3805566806 wait complete. 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 Processor -- start 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 -- start start 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 0x7f41cc1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 0x7f41cc19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1145f0 con 0x7f41cc10e8b0 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc114470 con 0x7f41cc074ee0 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.088+0000 7f41d0c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc114770 con 0x7f41cc0745c0 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51794/0 (socket says 172.21.15.123:51794) 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 -- 172.21.15.123:0/2300261797 learned_addr learned my addr 172.21.15.123:0/2300261797 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41cbe00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 0x7f41cc19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41cb400700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 0x7f41cc1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 -- 172.21.15.123:0/2300261797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 msgr2=0x7f41cc1a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 0x7f41cc1a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 -- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 msgr2=0x7f41cc19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 0x7f41cc19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 -- 172.21.15.123:0/2300261797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc19b010 con 0x7f41cc074ee0 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c3e00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41b4000c00 tx=0x7f41b400ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.089+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4027070 con 0x7f41cc074ee0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.090+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41b4003d70 con 0x7f41cc074ee0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.090+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc19b2a0 con 0x7f41cc074ee0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.090+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b400f460 con 0x7f41cc074ee0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.090+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc1a75c0 con 0x7f41cc074ee0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.091+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41cc068e50 con 0x7f41cc074ee0 2024-03-24T00:37:06.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.093+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f41b402a070 con 0x7f41cc074ee0 2024-03-24T00:37:06.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.094+0000 7f41c8c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 0x7f41b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.094+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41b40cf970 con 0x7f41cc074ee0 2024-03-24T00:37:06.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.094+0000 7f41cb400700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 0x7f41b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.095+0000 7f41cb400700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 0x7f41b00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41cc19b9f0 tx=0x7f41bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.099+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41b405e740 con 0x7f41cc074ee0 2024-03-24T00:37:06.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:06 smithi123 ceph-mon[31487]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.372+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f41cc02da30 con 0x7f41cc074ee0 2024-03-24T00:37:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.372+0000 7f41c8c00700 1 -- 172.21.15.123:0/2300261797 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f41b4013070 con 0x7f41cc074ee0 2024-03-24T00:37:06.373 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:06.373 INFO:teuthology.orchestra.run.smithi123.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:37:06.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.375+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 msgr2=0x7f41b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.375+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 0x7f41b00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41cc19b9f0 tx=0x7f41bc009260 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.375+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 msgr2=0x7f41cc1a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.375+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41b4000c00 tx=0x7f41b400ba90 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 shutdown_connections 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f41b00a2890 0x7f41b00a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc10e8b0 0x7f41cc19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41cc0745c0 0x7f41cc1a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 --2- 172.21.15.123:0/2300261797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41cc074ee0 0x7f41cc1a0c30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 >> 172.21.15.123:0/2300261797 conn(0x7f41cc0fdb60 msgr2=0x7f41cc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 shutdown_connections 2024-03-24T00:37:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:06.376+0000 7f41d0c00700 1 -- 172.21.15.123:0/2300261797 wait complete. 2024-03-24T00:37:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:06 smithi204 ceph-mon[36345]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:06 smithi177 ceph-mon[36034]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:07.223 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T00:37:07.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.445+0000 7fc897e00640 1 -- 172.21.15.123:0/599337553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc898107b00 msgr2=0x7fc898107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.445+0000 7fc897e00640 1 --2- 172.21.15.123:0/599337553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc898107b00 0x7fc898107f20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc888009860 tx=0x7fc88802bb70 comp rx=0 tx=0).stop 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 -- 172.21.15.123:0/599337553 shutdown_connections 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 --2- 172.21.15.123:0/599337553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8981096b0 0x7fc89810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 --2- 172.21.15.123:0/599337553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 0x7fc898109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 --2- 172.21.15.123:0/599337553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc898107b00 0x7fc898107f20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 -- 172.21.15.123:0/599337553 >> 172.21.15.123:0/599337553 conn(0x7fc898076ab0 msgr2=0x7fc898078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 -- 172.21.15.123:0/599337553 shutdown_connections 2024-03-24T00:37:07.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.446+0000 7fc897e00640 1 -- 172.21.15.123:0/599337553 wait complete. 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.447+0000 7fc897e00640 1 Processor -- start 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.447+0000 7fc897e00640 1 -- start start 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.447+0000 7fc897e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 0x7fc89819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8981096b0 0x7fc8981a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898113770 con 0x7fc898107b00 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981135f0 con 0x7fc8981096b0 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981138f0 con 0x7fc898108cf0 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52822/0 (socket says 172.21.15.123:52822) 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 -- 172.21.15.123:0/2228443584 learned_addr learned my addr 172.21.15.123:0/2228443584 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:07.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc897400640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8981096b0 0x7fc8981a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc88fe00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 0x7fc89819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 -- 172.21.15.123:0/2228443584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 msgr2=0x7fc89819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 0x7fc89819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 -- 172.21.15.123:0/2228443584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8981096b0 msgr2=0x7fc8981a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8981096b0 0x7fc8981a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.448+0000 7fc896a00640 1 -- 172.21.15.123:0/2228443584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8981a5a40 con 0x7fc898107b00 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc896a00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc8880349e0 tx=0x7fc888032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888041020 con 0x7fc898107b00 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc88802c9f0 con 0x7fc898107b00 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8981a5cd0 con 0x7fc898107b00 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888039040 con 0x7fc898107b00 2024-03-24T00:37:07.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.449+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8981a61c0 con 0x7fc898107b00 2024-03-24T00:37:07.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.450+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89810eea0 con 0x7fc898107b00 2024-03-24T00:37:07.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.452+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc888044020 con 0x7fc898107b00 2024-03-24T00:37:07.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.453+0000 7fc88f400640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 0x7fc87c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.454+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc8880f03a0 con 0x7fc898107b00 2024-03-24T00:37:07.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.454+0000 7fc88fe00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 0x7fc87c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.454+0000 7fc88fe00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 0x7fc87c0a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc89819f330 tx=0x7fc880005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.458+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc88804b020 con 0x7fc898107b00 2024-03-24T00:37:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:07 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2300261797' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:37:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:07 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2300261797' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:37:07.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.660+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc89802d290 con 0x7fc898107b00 2024-03-24T00:37:07.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.664+0000 7fc88f400640 1 -- 172.21.15.123:0/2228443584 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7fc8880bd3f0 con 0x7fc898107b00 2024-03-24T00:37:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:07.666 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 msgr2=0x7fc87c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 0x7fc87c0a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc89819f330 tx=0x7fc880005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 msgr2=0x7fc89819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc8880349e0 tx=0x7fc888032430 comp rx=0 tx=0).stop 2024-03-24T00:37:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 shutdown_connections 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8981096b0 0x7fc8981a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc898108cf0 0x7fc89819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc87c0a2800 0x7fc87c0a4cc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 --2- 172.21.15.123:0/2228443584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc898107b00 0x7fc89819de20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 >> 172.21.15.123:0/2228443584 conn(0x7fc898076ab0 msgr2=0x7fc89810c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.667+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 shutdown_connections 2024-03-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.668+0000 7fc897e00640 1 -- 172.21.15.123:0/2228443584 wait complete. 2024-03-24T00:37:07.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:07.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:07 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2300261797' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 -- 172.21.15.123:0/598145133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb4104ab0 msgr2=0x7feeb4104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 --2- 172.21.15.123:0/598145133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb4104ab0 0x7feeb4104f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feea0009900 tx=0x7feea002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 -- 172.21.15.123:0/598145133 shutdown_connections 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 --2- 172.21.15.123:0/598145133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 0x7feeb410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 --2- 172.21.15.123:0/598145133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb4104ab0 0x7feeb4104f30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 --2- 172.21.15.123:0/598145133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb41038c0 0x7feeb4103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 -- 172.21.15.123:0/598145133 >> 172.21.15.123:0/598145133 conn(0x7feeb40fee50 msgr2=0x7feeb4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 -- 172.21.15.123:0/598145133 shutdown_connections 2024-03-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.907+0000 7feeba200640 1 -- 172.21.15.123:0/598145133 wait complete. 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.908+0000 7feeba200640 1 Processor -- start 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.908+0000 7feeba200640 1 -- start start 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 0x7feeb419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 0x7feeb41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb410f590 con 0x7feeb4105470 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb410f410 con 0x7feeb41038c0 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeba200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb410f710 con 0x7feeb4104ab0 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51832/0 (socket says 172.21.15.123:51832) 2024-03-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 -- 172.21.15.123:0/1034320098 learned_addr learned my addr 172.21.15.123:0/1034320098 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3e00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 0x7feeb41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb2a00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 0x7feeb419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 -- 172.21.15.123:0/1034320098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 msgr2=0x7feeb419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 0x7feeb419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 -- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 msgr2=0x7feeb41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 0x7feeb41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb3400640 1 -- 172.21.15.123:0/1034320098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb41a1830 con 0x7feeb41038c0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.909+0000 7feeb2a00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 0x7feeb419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeb3400640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9c0 tx=0x7feea400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400ccc0 con 0x7feeb41038c0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea40035f0 con 0x7feeb41038c0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb41a1b20 con 0x7feeb41038c0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400ce60 con 0x7feeb41038c0 2024-03-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.910+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb41a2020 con 0x7feeb41038c0 2024-03-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.911+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb410abe0 con 0x7feeb41038c0 2024-03-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.913+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feea40037a0 con 0x7feeb41038c0 2024-03-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.914+0000 7feeabe00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 0x7fee980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.914+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feea40cef60 con 0x7feeb41038c0 2024-03-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.914+0000 7feeb2a00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 0x7fee980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.915+0000 7feeb2a00640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 0x7fee980a4cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feea002c240 tx=0x7feea002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:07.918+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feea409bfb0 con 0x7feeb41038c0 2024-03-24T00:37:08.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.117+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feeb402d070 con 0x7feeb41038c0 2024-03-24T00:37:08.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.118+0000 7feeabe00640 1 -- 172.21.15.123:0/1034320098 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7feea409b8b0 con 0x7feeb41038c0 2024-03-24T00:37:08.118 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 msgr2=0x7fee980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 0x7fee980a4cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feea002c240 tx=0x7feea002df90 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 msgr2=0x7feeb4199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9c0 tx=0x7feea400edc0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 shutdown_connections 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee980a2800 0x7fee980a4cc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feeb4105470 0x7feeb41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feeb4104ab0 0x7feeb419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 --2- 172.21.15.123:0/1034320098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb41038c0 0x7feeb4199c20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 >> 172.21.15.123:0/1034320098 conn(0x7feeb40fee50 msgr2=0x7feeb4107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 shutdown_connections 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.121+0000 7feeba200640 1 -- 172.21.15.123:0/1034320098 wait complete. 2024-03-24T00:37:08.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 -- 172.21.15.123:0/2150637935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2041028a0 msgr2=0x7fa204102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 --2- 172.21.15.123:0/2150637935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2041028a0 0x7fa204102d20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009900 tx=0x7fa1f002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 -- 172.21.15.123:0/2150637935 shutdown_connections 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 --2- 172.21.15.123:0/2150637935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa204103260 0x7fa204109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 --2- 172.21.15.123:0/2150637935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2041028a0 0x7fa204102d20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 --2- 172.21.15.123:0/2150637935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041016b0 0x7fa204101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 -- 172.21.15.123:0/2150637935 >> 172.21.15.123:0/2150637935 conn(0x7fa2040fcc40 msgr2=0x7fa2040ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 -- 172.21.15.123:0/2150637935 shutdown_connections 2024-03-24T00:37:08.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.354+0000 7fa20c600640 1 -- 172.21.15.123:0/2150637935 wait complete. 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.355+0000 7fa20c600640 1 Processor -- start 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.355+0000 7fa20c600640 1 -- start start 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 0x7fa204075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041028a0 0x7fa204073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20410d380 con 0x7fa2041028a0 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20410d200 con 0x7fa204103260 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20410d500 con 0x7fa2041016b0 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa209200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041028a0 0x7fa204073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51848/0 (socket says 172.21.15.123:51848) 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 -- 172.21.15.123:0/3197339835 learned_addr learned my addr 172.21.15.123:0/3197339835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa209c00640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 0x7fa204075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 -- 172.21.15.123:0/3197339835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 msgr2=0x7fa204075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 0x7fa204075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 -- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041028a0 msgr2=0x7fa204073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041028a0 0x7fa204073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa20a600640 1 -- 172.21.15.123:0/3197339835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204074860 con 0x7fa204103260 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.356+0000 7fa209c00640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 0x7fa204075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa20a600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00eb10 tx=0x7fa1fc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc004b20 con 0x7fa204103260 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1fc004cc0 con 0x7fa204103260 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc010760 con 0x7fa204103260 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2041a1ae0 con 0x7fa204103260 2024-03-24T00:37:08.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.357+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2041a1fb0 con 0x7fa204103260 2024-03-24T00:37:08.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.358+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2040688e0 con 0x7fa204103260 2024-03-24T00:37:08.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.360+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa1fc005020 con 0x7fa204103260 2024-03-24T00:37:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.361+0000 7fa202000640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 0x7fa1ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.361+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1fc0cf200 con 0x7fa204103260 2024-03-24T00:37:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.361+0000 7fa209c00640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 0x7fa1ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.362+0000 7fa209c00640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 0x7fa1ec0a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa204102700 tx=0x7fa1f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.365+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1fc09c250 con 0x7fa204103260 2024-03-24T00:37:08.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.563+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2040622a0 con 0x7fa204103260 2024-03-24T00:37:08.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.564+0000 7fa202000640 1 -- 172.21.15.123:0/3197339835 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1fc09bb50 con 0x7fa204103260 2024-03-24T00:37:08.565 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 msgr2=0x7fa1ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 0x7fa1ec0a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa204102700 tx=0x7fa1f8005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 msgr2=0x7fa204073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00eb10 tx=0x7fa1fc00c2d0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 shutdown_connections 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa204103260 0x7fa204073fd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1ec0a2880 0x7fa1ec0a4d40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2041028a0 0x7fa204073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 --2- 172.21.15.123:0/3197339835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2041016b0 0x7fa204075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 >> 172.21.15.123:0/3197339835 conn(0x7fa2040fcc40 msgr2=0x7fa204105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 shutdown_connections 2024-03-24T00:37:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.567+0000 7fa20c600640 1 -- 172.21.15.123:0/3197339835 wait complete. 2024-03-24T00:37:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:08 smithi204 ceph-mon[36345]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:08 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2228443584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:08 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1034320098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:08.584 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:08 smithi123 ceph-mon[31487]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:08.584 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:08 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2228443584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:08.584 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:08 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1034320098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:08.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:08 smithi177 ceph-mon[36034]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:08 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2228443584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:08 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1034320098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:08.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 -- 172.21.15.123:0/1796011181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 msgr2=0x7f40141019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 --2- 172.21.15.123:0/1796011181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f40141019a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4004009900 tx=0x7f400402bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 -- 172.21.15.123:0/1796011181 shutdown_connections 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 --2- 172.21.15.123:0/1796011181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014103130 0x7f40141099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 --2- 172.21.15.123:0/1796011181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014102770 0x7f4014102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 --2- 172.21.15.123:0/1796011181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f40141019a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 -- 172.21.15.123:0/1796011181 >> 172.21.15.123:0/1796011181 conn(0x7f40140fcb10 msgr2=0x7f40140fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 -- 172.21.15.123:0/1796011181 shutdown_connections 2024-03-24T00:37:08.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.806+0000 7f4013e00640 1 -- 172.21.15.123:0/1796011181 wait complete. 2024-03-24T00:37:08.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.807+0000 7f4013e00640 1 Processor -- start 2024-03-24T00:37:08.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.807+0000 7f4013e00640 1 -- start start 2024-03-24T00:37:08.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014102770 0x7f4014199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014103130 0x7f40141a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401410d240 con 0x7f4014102770 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401410d0c0 con 0x7f4014101580 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401410d3c0 con 0x7f4014103130 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014102770 0x7f4014199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51860/0 (socket says 172.21.15.123:51860) 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 -- 172.21.15.123:0/3809053158 learned_addr learned my addr 172.21.15.123:0/3809053158 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4013400640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014103130 0x7f40141a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 -- 172.21.15.123:0/3809053158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014103130 msgr2=0x7f40141a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014103130 0x7f40141a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 -- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014102770 msgr2=0x7f4014199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014102770 0x7f4014199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.808+0000 7f4012a00640 1 -- 172.21.15.123:0/3809053158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40141a16a0 con 0x7f4014101580 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4012a00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f400402efd0 tx=0x7f4004005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004041070 con 0x7f4014101580 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40040054e0 con 0x7f4014101580 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004034440 con 0x7f4014101580 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40141a1930 con 0x7f4014101580 2024-03-24T00:37:08.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.809+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40141a1df0 con 0x7f4014101580 2024-03-24T00:37:08.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.810+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40141a1ff0 con 0x7f4014101580 2024-03-24T00:37:08.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.812+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4004044070 con 0x7f4014101580 2024-03-24T00:37:08.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.813+0000 7f4018a00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 0x7f3ff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.813+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f40040f0690 con 0x7f4014101580 2024-03-24T00:37:08.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.813+0000 7f4012000640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 0x7f3ff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.814+0000 7f4012000640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 0x7f3ff80a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f401419af90 tx=0x7f400000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:08.818+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f40040bd0f0 con 0x7f4014101580 2024-03-24T00:37:09.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.021+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40141a22e0 con 0x7f4014101580 2024-03-24T00:37:09.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.022+0000 7f4018a00640 1 -- 172.21.15.123:0/3809053158 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40040c1190 con 0x7f4014101580 2024-03-24T00:37:09.022 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 msgr2=0x7f3ff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 0x7f3ff80a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f401419af90 tx=0x7f400000b390 comp rx=0 tx=0).stop 2024-03-24T00:37:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 msgr2=0x7f4014199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f400402efd0 tx=0x7f4004005e40 comp rx=0 tx=0).stop 2024-03-24T00:37:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 shutdown_connections 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4014103130 0x7f40141a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff80a2800 0x7f3ff80a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4014102770 0x7f4014199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 --2- 172.21.15.123:0/3809053158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101580 0x7f4014199a80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 >> 172.21.15.123:0/3809053158 conn(0x7f40140fcb10 msgr2=0x7f40140738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.024+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 shutdown_connections 2024-03-24T00:37:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.025+0000 7f4013e00640 1 -- 172.21.15.123:0/3809053158 wait complete. 2024-03-24T00:37:09.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 -- 172.21.15.123:0/640380692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0756e0 msgr2=0x7fad5c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 --2- 172.21.15.123:0/640380692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0756e0 0x7fad5c073b30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009900 tx=0x7fad4c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 -- 172.21.15.123:0/640380692 shutdown_connections 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 --2- 172.21.15.123:0/640380692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c109580 0x7fad5c10ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 --2- 172.21.15.123:0/640380692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c074100 0x7fad5c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 --2- 172.21.15.123:0/640380692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0756e0 0x7fad5c073b30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.253+0000 7fad62400640 1 -- 172.21.15.123:0/640380692 >> 172.21.15.123:0/640380692 conn(0x7fad5c0fcc50 msgr2=0x7fad5c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.254+0000 7fad62400640 1 -- 172.21.15.123:0/640380692 shutdown_connections 2024-03-24T00:37:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.254+0000 7fad62400640 1 -- 172.21.15.123:0/640380692 wait complete. 2024-03-24T00:37:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 Processor -- start 2024-03-24T00:37:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 -- start start 2024-03-24T00:37:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 0x7fad5c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c109580 0x7fad5c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c10f550 con 0x7fad5c109580 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c10f3d0 con 0x7fad5c074100 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.255+0000 7fad62400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c10f6d0 con 0x7fad5c0756e0 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51872/0 (socket says 172.21.15.123:51872) 2024-03-24T00:37:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 -- 172.21.15.123:0/4235545982 learned_addr learned my addr 172.21.15.123:0/4235545982 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5be00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c109580 0x7fad5c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5aa00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 0x7fad5c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 -- 172.21.15.123:0/4235545982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 msgr2=0x7fad5c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 0x7fad5c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 -- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c109580 msgr2=0x7fad5c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c109580 0x7fad5c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 -- 172.21.15.123:0/4235545982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c1a1800 con 0x7fad5c074100 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5aa00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 0x7fad5c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.256+0000 7fad5b400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005ab0 tx=0x7fad4c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.257+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c043020 con 0x7fad5c074100 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.257+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad4c0054e0 con 0x7fad5c074100 2024-03-24T00:37:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.257+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c1a1a90 con 0x7fad5c074100 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.257+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c003ce0 con 0x7fad5c074100 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.257+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c1a1fe0 con 0x7fad5c074100 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.258+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c1082b0 con 0x7fad5c074100 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.260+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fad4c046020 con 0x7fad5c074100 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.261+0000 7fad53e00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 0x7fad400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.261+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fad4c0f0bd0 con 0x7fad5c074100 2024-03-24T00:37:09.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.261+0000 7fad5aa00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 0x7fad400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.262+0000 7fad5aa00640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 0x7fad400a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad5c19b0f0 tx=0x7fad48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.265+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fad4c04d020 con 0x7fad5c074100 2024-03-24T00:37:09.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:09 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3197339835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:09 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3809053158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.507+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5c0622a0 con 0x7fad5c074100 2024-03-24T00:37:09.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.508+0000 7fad53e00640 1 -- 172.21.15.123:0/4235545982 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fad4c0bdc80 con 0x7fad5c074100 2024-03-24T00:37:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:09.509 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:09.509 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 msgr2=0x7fad400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 0x7fad400a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad5c19b0f0 tx=0x7fad48005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 msgr2=0x7fad5c199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005ab0 tx=0x7fad4c02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 shutdown_connections 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad400a2800 0x7fad400a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad5c109580 0x7fad5c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad5c0756e0 0x7fad5c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 --2- 172.21.15.123:0/4235545982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c074100 0x7fad5c199be0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 >> 172.21.15.123:0/4235545982 conn(0x7fad5c0fcc50 msgr2=0x7fad5c105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 shutdown_connections 2024-03-24T00:37:09.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.511+0000 7fad62400640 1 -- 172.21.15.123:0/4235545982 wait complete. 2024-03-24T00:37:09.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:09 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3197339835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:09 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3809053158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:09 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3197339835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:09 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3809053158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 -- 172.21.15.123:0/351648759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 msgr2=0x7f3fc010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/351648759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc010ff40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009900 tx=0x7f3fb802bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 -- 172.21.15.123:0/351648759 shutdown_connections 2024-03-24T00:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/351648759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc010ff40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/351648759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc0109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/351648759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 0x7f3fc0107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 -- 172.21.15.123:0/351648759 >> 172.21.15.123:0/351648759 conn(0x7f3fc0076ab0 msgr2=0x7f3fc0078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 -- 172.21.15.123:0/351648759 shutdown_connections 2024-03-24T00:37:09.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.743+0000 7f3fc5a00640 1 -- 172.21.15.123:0/351648759 wait complete. 2024-03-24T00:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.744+0000 7f3fc5a00640 1 Processor -- start 2024-03-24T00:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.744+0000 7f3fc5a00640 1 -- start start 2024-03-24T00:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 0x7f3fc019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01137d0 con 0x7f3fc0108cf0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0113650 con 0x7f3fc01096b0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0113950 con 0x7f3fc0107b00 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51904/0 (socket says 172.21.15.123:51904) 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 -- 172.21.15.123:0/665820128 learned_addr learned my addr 172.21.15.123:0/665820128 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc019e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52922/0 (socket says 172.21.15.123:52922) 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fbfe00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 0x7f3fc019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 -- 172.21.15.123:0/665820128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 msgr2=0x7f3fc019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 0x7f3fc019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 -- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 msgr2=0x7f3fc019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 -- 172.21.15.123:0/665820128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc01a5980 con 0x7f3fc01096b0 2024-03-24T00:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.745+0000 7f3fc5000640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3fb802c8e0 tx=0x7f3fb802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.746+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8043020 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.746+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb80054e0 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.746+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc01a5c10 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.746+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8003d00 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.746+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc01a6160 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.747+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc010ed50 con 0x7f3fc01096b0 2024-03-24T00:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.749+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3fb8046020 con 0x7f3fc01096b0 2024-03-24T00:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.750+0000 7f3fbcc00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 0x7f3fa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.750+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3fb80f0b00 con 0x7f3fc01096b0 2024-03-24T00:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.750+0000 7f3fbfe00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 0x7f3fa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.751+0000 7f3fbfe00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 0x7f3fa40a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0108b50 tx=0x7f3fb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.754+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3fb804d020 con 0x7f3fc01096b0 2024-03-24T00:37:09.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.956+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fc00622a0 con 0x7f3fc01096b0 2024-03-24T00:37:09.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.958+0000 7f3fbcc00640 1 -- 172.21.15.123:0/665820128 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f3fb80bdbe0 con 0x7f3fc01096b0 2024-03-24T00:37:09.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:09.959 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 msgr2=0x7f3fa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 0x7f3fa40a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0108b50 tx=0x7f3fb0009500 comp rx=0 tx=0).stop 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 msgr2=0x7f3fc01a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3fb802c8e0 tx=0x7f3fb802c640 comp rx=0 tx=0).stop 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 shutdown_connections 2024-03-24T00:37:09.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fc01096b0 0x7f3fc01a5310 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3fa40a2800 0x7f3fa40a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fc0108cf0 0x7f3fc019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 --2- 172.21.15.123:0/665820128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fc0107b00 0x7f3fc019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 >> 172.21.15.123:0/665820128 conn(0x7f3fc0076ab0 msgr2=0x7f3fc010bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 shutdown_connections 2024-03-24T00:37:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:09.961+0000 7f3fc5a00640 1 -- 172.21.15.123:0/665820128 wait complete. 2024-03-24T00:37:09.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:09.981 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 -- 172.21.15.123:0/1635565359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf4102150 msgr2=0x7f3bf41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 --2- 172.21.15.123:0/1635565359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf4102150 0x7f3bf41089e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009860 tx=0x7f3bec02bb70 comp rx=0 tx=0).stop 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 -- 172.21.15.123:0/1635565359 shutdown_connections 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 --2- 172.21.15.123:0/1635565359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf4102150 0x7f3bf41089e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 --2- 172.21.15.123:0/1635565359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 0x7f3bf4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 --2- 172.21.15.123:0/1635565359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf41005a0 0x7f3bf41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 -- 172.21.15.123:0/1635565359 >> 172.21.15.123:0/1635565359 conn(0x7f3bf40fbb30 msgr2=0x7f3bf40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 -- 172.21.15.123:0/1635565359 shutdown_connections 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.196+0000 7f3bfc600640 1 -- 172.21.15.123:0/1635565359 wait complete. 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 Processor -- start 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 -- start start 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 0x7f3bf4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf4102150 0x7f3bf419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf410c230 con 0x7f3bf4102150 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf410c0b0 con 0x7f3bf41005a0 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.197+0000 7f3bfc600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf410c3b0 con 0x7f3bf4101790 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51926/0 (socket says 172.21.15.123:51926) 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 -- 172.21.15.123:0/553583254 learned_addr learned my addr 172.21.15.123:0/553583254 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bfa600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf4102150 0x7f3bf419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9200640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 0x7f3bf4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 -- 172.21.15.123:0/553583254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 msgr2=0x7f3bf4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 0x7f3bf4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 -- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf4102150 msgr2=0x7f3bf419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf4102150 0x7f3bf419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 -- 172.21.15.123:0/553583254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf419f610 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf9c00640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3be800c3f0 tx=0x7f3be8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be801b070 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3be800bb10 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8016d80 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.198+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf419f900 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.199+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf419fe50 con 0x7f3bf41005a0 2024-03-24T00:37:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.200+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf41a00e0 con 0x7f3bf41005a0 2024-03-24T00:37:10.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.202+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3be801e070 con 0x7f3bf41005a0 2024-03-24T00:37:10.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.203+0000 7f3bf2000640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 0x7f3bdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.203+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3be80cecd0 con 0x7f3bf41005a0 2024-03-24T00:37:10.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.203+0000 7f3bf9200640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 0x7f3bdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.204+0000 7f3bf9200640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 0x7f3bdc0a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4198f00 tx=0x7f3be0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.205+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3be809bd20 con 0x7f3bf41005a0 2024-03-24T00:37:10.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.401+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bf41a02b0 con 0x7f3bf41005a0 2024-03-24T00:37:10.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.402+0000 7f3bf2000640 1 -- 172.21.15.123:0/553583254 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3be809b620 con 0x7f3bf41005a0 2024-03-24T00:37:10.403 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:10.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.405+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 msgr2=0x7f3bdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.405+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 0x7f3bdc0a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4198f00 tx=0x7f3be0005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.405+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 msgr2=0x7f3bf4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.405+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3be800c3f0 tx=0x7f3be8009f90 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 shutdown_connections 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3bdc0a2880 0x7f3bdc0a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bf4102150 0x7f3bf419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3bf4101790 0x7f3bf4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 --2- 172.21.15.123:0/553583254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bf41005a0 0x7f3bf4197a60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 >> 172.21.15.123:0/553583254 conn(0x7f3bf40fbb30 msgr2=0x7f3bf40fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 shutdown_connections 2024-03-24T00:37:10.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.406+0000 7f3bfc600640 1 -- 172.21.15.123:0/553583254 wait complete. 2024-03-24T00:37:10.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:10 smithi204 ceph-mon[36345]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:10 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4235545982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:10 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/665820128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:10 smithi123 ceph-mon[31487]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:10 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4235545982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:10 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/665820128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.637+0000 7f1db5000640 1 -- 172.21.15.123:0/1240712783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 msgr2=0x7f1db0077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.637+0000 7f1db5000640 1 --2- 172.21.15.123:0/1240712783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db0077700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009900 tx=0x7f1d9c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 -- 172.21.15.123:0/1240712783 shutdown_connections 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 --2- 172.21.15.123:0/1240712783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 --2- 172.21.15.123:0/1240712783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db0077700 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 --2- 172.21.15.123:0/1240712783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 0x7f1db0076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 -- 172.21.15.123:0/1240712783 >> 172.21.15.123:0/1240712783 conn(0x7f1db00fee50 msgr2=0x7f1db0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 -- 172.21.15.123:0/1240712783 shutdown_connections 2024-03-24T00:37:10.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.638+0000 7f1db5000640 1 -- 172.21.15.123:0/1240712783 wait complete. 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 Processor -- start 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 -- start start 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 0x7f1db019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db01a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01139b0 con 0x7f1db0077c40 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0113830 con 0x7f1db00772a0 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.639+0000 7f1db5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0113b30 con 0x7f1db0076060 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dae800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db01a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51948/0 (socket says 172.21.15.123:51948) 2024-03-24T00:37:10.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 -- 172.21.15.123:0/1752629081 learned_addr learned my addr 172.21.15.123:0/1752629081 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dade00640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 0x7f1db019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 -- 172.21.15.123:0/1752629081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 msgr2=0x7f1db019dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 0x7f1db019dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 -- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 msgr2=0x7f1db01a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db01a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 -- 172.21.15.123:0/1752629081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db01a5c10 con 0x7f1db00772a0 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dae800640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db01a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:10.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.640+0000 7f1dad400640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c000c00 tx=0x7f1d9c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.641+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c043020 con 0x7f1db00772a0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.641+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d9c0054e0 con 0x7f1db00772a0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.641+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c003c40 con 0x7f1db00772a0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.641+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db01a5ea0 con 0x7f1db00772a0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.641+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db01a6310 con 0x7f1db00772a0 2024-03-24T00:37:10.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.642+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db0110250 con 0x7f1db00772a0 2024-03-24T00:37:10.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.644+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1d9c046020 con 0x7f1db00772a0 2024-03-24T00:37:10.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.645+0000 7f1da6a00640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 0x7f1d940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.645+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1d9c0f0a70 con 0x7f1db00772a0 2024-03-24T00:37:10.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.645+0000 7f1dade00640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 0x7f1d940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.646+0000 7f1dade00640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 0x7f1d940a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009bd0 tx=0x7f1da0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.649+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d9c04d020 con 0x7f1db00772a0 2024-03-24T00:37:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:10 smithi177 ceph-mon[36034]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:10 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4235545982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:10 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/665820128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:10.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.847+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1db00622a0 con 0x7f1db00772a0 2024-03-24T00:37:10.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.847+0000 7f1da6a00640 1 -- 172.21.15.123:0/1752629081 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1d9c0bdac0 con 0x7f1db00772a0 2024-03-24T00:37:10.848 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 msgr2=0x7f1d940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 0x7f1d940a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009bd0 tx=0x7f1da0005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 msgr2=0x7f1db019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c000c00 tx=0x7f1d9c02c650 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 shutdown_connections 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1d940a2880 0x7f1d940a4d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1db0077c40 0x7f1db01a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db00772a0 0x7f1db019e530 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 --2- 172.21.15.123:0/1752629081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1db0076060 0x7f1db019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 >> 172.21.15.123:0/1752629081 conn(0x7f1db00fee50 msgr2=0x7f1db0100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 shutdown_connections 2024-03-24T00:37:10.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:10.850+0000 7f1db5000640 1 -- 172.21.15.123:0/1752629081 wait complete. 2024-03-24T00:37:10.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:11.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 -- 172.21.15.123:0/519063934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 msgr2=0x7fbe38103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/519063934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38103d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009ca0 tx=0x7fbe2802c100 comp rx=0 tx=0).stop 2024-03-24T00:37:11.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 -- 172.21.15.123:0/519063934 shutdown_connections 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/519063934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 0x7fbe3810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/519063934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe38104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/519063934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38103d00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 -- 172.21.15.123:0/519063934 >> 172.21.15.123:0/519063934 conn(0x7fbe380fee50 msgr2=0x7fbe38101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 -- 172.21.15.123:0/519063934 shutdown_connections 2024-03-24T00:37:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.076+0000 7fbe3ce00640 1 -- 172.21.15.123:0/519063934 wait complete. 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 Processor -- start 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 -- start start 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe3819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 0x7fbe381a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3810f560 con 0x7fbe38104ad0 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3810f3e0 con 0x7fbe381038e0 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe3ce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3810f6e0 con 0x7fbe38105490 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe2d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe3819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe35e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe35e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51962/0 (socket says 172.21.15.123:51962) 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe2d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe3819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52976/0 (socket says 172.21.15.123:52976) 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.077+0000 7fbe35e00640 1 -- 172.21.15.123:0/3207559635 learned_addr learned my addr 172.21.15.123:0/3207559635 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe36800640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 0x7fbe381a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 -- 172.21.15.123:0/3207559635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 msgr2=0x7fbe381a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 0x7fbe381a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 -- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 msgr2=0x7fbe3819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe3819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 -- 172.21.15.123:0/3207559635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe381a1830 con 0x7fbe381038e0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe35e00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f80 tx=0x7fbe28035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28039040 con 0x7fbe381038e0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe28044070 con 0x7fbe381038e0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004430 con 0x7fbe381038e0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.078+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe381a1ac0 con 0x7fbe381038e0 2024-03-24T00:37:11.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.079+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe381a1f30 con 0x7fbe381038e0 2024-03-24T00:37:11.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.080+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3810ac50 con 0x7fbe381038e0 2024-03-24T00:37:11.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.081+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbe280045d0 con 0x7fbe381038e0 2024-03-24T00:37:11.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.083+0000 7fbe2f400640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 0x7fbe1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.083+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbe280f0ff0 con 0x7fbe381038e0 2024-03-24T00:37:11.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.083+0000 7fbe2d400640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 0x7fbe1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.084+0000 7fbe2d400640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 0x7fbe1c0a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbe200095f0 tx=0x7fbe20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:11.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.087+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe280be080 con 0x7fbe381038e0 2024-03-24T00:37:11.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.288+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe380622d0 con 0x7fbe381038e0 2024-03-24T00:37:11.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.293+0000 7fbe2f400640 1 -- 172.21.15.123:0/3207559635 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe280bd980 con 0x7fbe381038e0 2024-03-24T00:37:11.294 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 msgr2=0x7fbe1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 0x7fbe1c0a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbe200095f0 tx=0x7fbe20005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 msgr2=0x7fbe38199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f80 tx=0x7fbe28035d10 comp rx=0 tx=0).stop 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 shutdown_connections 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe38105490 0x7fbe381a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbe1c0a2880 0x7fbe1c0a4d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe38104ad0 0x7fbe3819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 --2- 172.21.15.123:0/3207559635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe381038e0 0x7fbe38199c20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 >> 172.21.15.123:0/3207559635 conn(0x7fbe380fee50 msgr2=0x7fbe38107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:11.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 shutdown_connections 2024-03-24T00:37:11.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.296+0000 7fbe3ce00640 1 -- 172.21.15.123:0/3207559635 wait complete. 2024-03-24T00:37:11.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 -- 172.21.15.123:0/4210635018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d4105490 msgr2=0x7f82d410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 --2- 172.21.15.123:0/4210635018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d4105490 0x7f82d410bd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009ca0 tx=0x7f82cc02c100 comp rx=0 tx=0).stop 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 -- 172.21.15.123:0/4210635018 shutdown_connections 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 --2- 172.21.15.123:0/4210635018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d4105490 0x7f82d410bd20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 --2- 172.21.15.123:0/4210635018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 0x7f82d4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 --2- 172.21.15.123:0/4210635018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d41038e0 0x7f82d4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.520+0000 7f82d9000640 1 -- 172.21.15.123:0/4210635018 >> 172.21.15.123:0/4210635018 conn(0x7f82d40fee50 msgr2=0x7f82d4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.521+0000 7f82d9000640 1 -- 172.21.15.123:0/4210635018 shutdown_connections 2024-03-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.521+0000 7f82d9000640 1 -- 172.21.15.123:0/4210635018 wait complete. 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.521+0000 7f82d9000640 1 Processor -- start 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 -- start start 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 0x7f82d419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d4105490 0x7f82d41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d410f560 con 0x7f82d4104ad0 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d410f3e0 con 0x7f82d41038e0 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d9000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d410f6e0 con 0x7f82d4105490 2024-03-24T00:37:11.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51984/0 (socket says 172.21.15.123:51984) 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d1e00640 1 -- 172.21.15.123:0/2490502230 learned_addr learned my addr 172.21.15.123:0/2490502230 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82c9400640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 0x7f82d419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.522+0000 7f82d2800640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d4105490 0x7f82d41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 -- 172.21.15.123:0/2490502230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d4105490 msgr2=0x7f82d41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d4105490 0x7f82d41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 -- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 msgr2=0x7f82d419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 0x7f82d419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 -- 172.21.15.123:0/2490502230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d41a17c0 con 0x7f82d41038e0 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d1e00640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f82c400e9c0 tx=0x7f82c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:11.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c400cd80 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82c40035f0 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c4010690 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d41a1ab0 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.523+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d41a1fb0 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.524+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d410aba0 con 0x7f82d41038e0 2024-03-24T00:37:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.526+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f82c40037a0 con 0x7f82d41038e0 2024-03-24T00:37:11.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.527+0000 7f82cb400640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 0x7f82b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.527+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f82c40cedb0 con 0x7f82d41038e0 2024-03-24T00:37:11.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.527+0000 7f82c9400640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 0x7f82b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.528+0000 7f82c9400640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 0x7f82b40a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82d419b0b0 tx=0x7f82bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:11.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.535+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82c409be60 con 0x7f82d41038e0 2024-03-24T00:37:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:11 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/553583254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:11 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1752629081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:11 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/553583254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:11 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1752629081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:11 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/553583254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:11 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1752629081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:11.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.771+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d402d070 con 0x7f82d41038e0 2024-03-24T00:37:11.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.773+0000 7f82cb400640 1 -- 172.21.15.123:0/2490502230 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f82c409b760 con 0x7f82d41038e0 2024-03-24T00:37:11.774 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:11.774 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:11.774 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 msgr2=0x7f82b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 0x7f82b40a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82d419b0b0 tx=0x7f82bc005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 msgr2=0x7f82d4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f82c400e9c0 tx=0x7f82c400ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 shutdown_connections 2024-03-24T00:37:11.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82d4105490 0x7f82d41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f82b40a2800 0x7f82b40a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82d4104ad0 0x7f82d419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 --2- 172.21.15.123:0/2490502230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82d41038e0 0x7f82d4199ba0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 >> 172.21.15.123:0/2490502230 conn(0x7f82d40fee50 msgr2=0x7f82d4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 shutdown_connections 2024-03-24T00:37:11.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.775+0000 7f82d9000640 1 -- 172.21.15.123:0/2490502230 wait complete. 2024-03-24T00:37:11.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:11.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.993+0000 7f3257e00640 1 -- 172.21.15.123:0/3354508684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250105490 msgr2=0x7f325010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.993+0000 7f3257e00640 1 --2- 172.21.15.123:0/3354508684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250105490 0x7f325010bd20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3248009be0 tx=0x7f324802bef0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.993+0000 7f3257e00640 1 -- 172.21.15.123:0/3354508684 shutdown_connections 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.993+0000 7f3257e00640 1 --2- 172.21.15.123:0/3354508684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250105490 0x7f325010bd20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.994+0000 7f3257e00640 1 --2- 172.21.15.123:0/3354508684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3250104ad0 0x7f3250104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.994+0000 7f3257e00640 1 --2- 172.21.15.123:0/3354508684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32501038e0 0x7f3250103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.994+0000 7f3257e00640 1 -- 172.21.15.123:0/3354508684 >> 172.21.15.123:0/3354508684 conn(0x7f32500fee50 msgr2=0x7f3250101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.994+0000 7f3257e00640 1 -- 172.21.15.123:0/3354508684 shutdown_connections 2024-03-24T00:37:11.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.994+0000 7f3257e00640 1 -- 172.21.15.123:0/3354508684 wait complete. 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 Processor -- start 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 -- start start 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250104ad0 0x7f325019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3250105490 0x7f32501a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325010f540 con 0x7f3250105490 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325010f3c0 con 0x7f32501038e0 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.995+0000 7f3257e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325010f6c0 con 0x7f3250104ad0 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3250105490 0x7f32501a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52008/0 (socket says 172.21.15.123:52008) 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 -- 172.21.15.123:0/2007927982 learned_addr learned my addr 172.21.15.123:0/2007927982 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3254a00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250104ad0 0x7f325019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 -- 172.21.15.123:0/2007927982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250104ad0 msgr2=0x7f325019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250104ad0 0x7f325019a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 -- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3250105490 msgr2=0x7f32501a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:11.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3250105490 0x7f32501a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:11.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 -- 172.21.15.123:0/2007927982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32501a17d0 con 0x7f32501038e0 2024-03-24T00:37:11.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3255400640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f324000e9c0 tx=0x7f324000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324000cd80 con 0x7f32501038e0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32400035f0 con 0x7f32501038e0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32501a1ac0 con 0x7f32501038e0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.996+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240010690 con 0x7f32501038e0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.997+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32501a2010 con 0x7f32501038e0 2024-03-24T00:37:11.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.998+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325010ac00 con 0x7f32501038e0 2024-03-24T00:37:12.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:11.999+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f32400037a0 con 0x7f32501038e0 2024-03-24T00:37:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.000+0000 7f324e000640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 0x7f32380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.001+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f32400cee00 con 0x7f32501038e0 2024-03-24T00:37:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.001+0000 7f3254a00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 0x7f32380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.001+0000 7f3254a00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 0x7f32380a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f325019b0c0 tx=0x7f3244009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.004+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f324009beb0 con 0x7f32501038e0 2024-03-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.209+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3250062340 con 0x7f32501038e0 2024-03-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.211+0000 7f324e000640 1 -- 172.21.15.123:0/2007927982 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f324009b7b0 con 0x7f32501038e0 2024-03-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 msgr2=0x7f32380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 0x7f32380a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f325019b0c0 tx=0x7f3244009260 comp rx=0 tx=0).stop 2024-03-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 msgr2=0x7f3250199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f324000e9c0 tx=0x7f324000ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 shutdown_connections 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f32380a2800 0x7f32380a4cc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3250105490 0x7f32501a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3250104ad0 0x7f325019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 --2- 172.21.15.123:0/2007927982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32501038e0 0x7f3250199bb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.214+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 >> 172.21.15.123:0/2007927982 conn(0x7f32500fee50 msgr2=0x7f3250107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.215+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 shutdown_connections 2024-03-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.215+0000 7f3257e00640 1 -- 172.21.15.123:0/2007927982 wait complete. 2024-03-24T00:37:12.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:12.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.430+0000 7f6205000640 1 -- 172.21.15.123:0/735936126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200104a70 msgr2=0x7f6200106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.430+0000 7f6205000640 1 --2- 172.21.15.123:0/735936126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200104a70 0x7f6200106e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009900 tx=0x7f61ec02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 -- 172.21.15.123:0/735936126 shutdown_connections 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 --2- 172.21.15.123:0/735936126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 0x7f62001097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 --2- 172.21.15.123:0/735936126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200104a70 0x7f6200106e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 --2- 172.21.15.123:0/735936126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200102140 0x7f6200104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 -- 172.21.15.123:0/735936126 >> 172.21.15.123:0/735936126 conn(0x7f62000fbb30 msgr2=0x7f62000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 -- 172.21.15.123:0/735936126 shutdown_connections 2024-03-24T00:37:12.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.431+0000 7f6205000640 1 -- 172.21.15.123:0/735936126 wait complete. 2024-03-24T00:37:12.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 Processor -- start 2024-03-24T00:37:12.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 -- start start 2024-03-24T00:37:12.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200104a70 0x7f6200197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 0x7f620019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620010d380 con 0x7f62001073a0 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620010d200 con 0x7f6200102140 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.432+0000 7f6205000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620010d500 con 0x7f6200104a70 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52022/0 (socket says 172.21.15.123:52022) 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 -- 172.21.15.123:0/1767739745 learned_addr learned my addr 172.21.15.123:0/1767739745 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fd400640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200104a70 0x7f6200197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fe800640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 0x7f620019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 -- 172.21.15.123:0/1767739745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200104a70 msgr2=0x7f6200197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200104a70 0x7f6200197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 -- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 msgr2=0x7f620019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 0x7f620019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 -- 172.21.15.123:0/1767739745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620019f5f0 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.433+0000 7f61fde00640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61f000d950 tx=0x7f61f000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.434+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0017030 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.434+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61f0004020 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.434+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620019f8e0 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.434+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f00047a0 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.434+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620019fe50 con 0x7f6200102140 2024-03-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.435+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6200068490 con 0x7f6200102140 2024-03-24T00:37:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.436+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f61f0009a30 con 0x7f6200102140 2024-03-24T00:37:12.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.438+0000 7f61f6a00640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 0x7f61e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.438+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f61f00ce3d0 con 0x7f6200102140 2024-03-24T00:37:12.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.438+0000 7f61fd400640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 0x7f61e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.439+0000 7f61fd400640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 0x7f61e40a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f61ec02c170 tx=0x7f61ec02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.442+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f61f009b480 con 0x7f6200102140 2024-03-24T00:37:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:12 smithi204 ceph-mon[36345]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:12 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3207559635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:12 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2490502230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:12 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2007927982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.595 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:12 smithi123 ceph-mon[31487]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:12.595 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:12 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3207559635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:12.595 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:12 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2490502230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.595 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:12 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2007927982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.699+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62000622a0 con 0x7f6200102140 2024-03-24T00:37:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.700+0000 7f61f6a00640 1 -- 172.21.15.123:0/1767739745 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f61f009ad80 con 0x7f6200102140 2024-03-24T00:37:12.702 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:12.702 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 msgr2=0x7f61e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 0x7f61e40a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f61ec02c170 tx=0x7f61ec02df90 comp rx=0 tx=0).stop 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 msgr2=0x7f62001979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61f000d950 tx=0x7f61f000de10 comp rx=0 tx=0).stop 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 shutdown_connections 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f61e40a2880 0x7f61e40a4d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62001073a0 0x7f620019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6200104a70 0x7f6200197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 --2- 172.21.15.123:0/1767739745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6200102140 0x7f62001979d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 >> 172.21.15.123:0/1767739745 conn(0x7f62000fbb30 msgr2=0x7f6200102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 shutdown_connections 2024-03-24T00:37:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.705+0000 7f6205000640 1 -- 172.21.15.123:0/1767739745 wait complete. 2024-03-24T00:37:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:12 smithi177 ceph-mon[36034]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:12 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3207559635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:12 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2490502230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:12 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2007927982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:12.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:35:32.796117+0000', 'modified': '2024-03-24T00:35:36.513248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24560': {'gid': 24560, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6841/829451039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6840', 'nonce': 829451039}, {'type': 'v1', 'addr': '172.21.15.123:6841', 'nonce': 829451039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6839/2781882004', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 2781882004}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 2781882004}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24572, 'qdb_cluster': [24572]} 2024-03-24T00:37:12.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:37:12.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 -- 172.21.15.123:0/1138509706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184107400 msgr2=0x7ff1841097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1138509706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184107400 0x7ff1841097f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009ca0 tx=0x7ff17c02c100 comp rx=0 tx=0).stop 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 -- 172.21.15.123:0/1138509706 shutdown_connections 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1138509706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184107400 0x7ff1841097f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1138509706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184104ad0 0x7ff184106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1138509706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.922+0000 7ff18ae00640 1 -- 172.21.15.123:0/1138509706 >> 172.21.15.123:0/1138509706 conn(0x7ff1840fbb50 msgr2=0x7ff1840fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.923+0000 7ff18ae00640 1 -- 172.21.15.123:0/1138509706 shutdown_connections 2024-03-24T00:37:12.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.923+0000 7ff18ae00640 1 -- 172.21.15.123:0/1138509706 wait complete. 2024-03-24T00:37:12.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.923+0000 7ff18ae00640 1 Processor -- start 2024-03-24T00:37:12.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 -- start start 2024-03-24T00:37:12.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184107400 0x7ff1841a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18410d340 con 0x7ff184107400 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18410d1c0 con 0x7ff184104ad0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff18ae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18410d4c0 con 0x7ff1841021a0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52036/0 (socket says 172.21.15.123:52036) 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 -- 172.21.15.123:0/1531353224 learned_addr learned my addr 172.21.15.123:0/1531353224 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183e00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff188e00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184107400 0x7ff1841a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 -- 172.21.15.123:0/1531353224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 msgr2=0x7ff184199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 -- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184107400 msgr2=0x7ff1841a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.924+0000 7ff183400640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184107400 0x7ff1841a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff183400640 1 -- 172.21.15.123:0/1531353224 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1841a1810 con 0x7ff184104ad0 2024-03-24T00:37:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff183e00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff183400640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff17000e9c0 tx=0x7ff17000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff180c00640 1 -- 172.21.15.123:0/1531353224 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17000cd80 con 0x7ff184104ad0 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff180c00640 1 -- 172.21.15.123:0/1531353224 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff170005500 con 0x7ff184104ad0 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1841a1b00 con 0x7ff184104ad0 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff180c00640 1 -- 172.21.15.123:0/1531353224 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170010690 con 0x7ff184104ad0 2024-03-24T00:37:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.925+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1841a20a0 con 0x7ff184104ad0 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.928+0000 7ff18dbaf740 1 Processor -- start 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 -- start start 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff180c00640 1 -- 172.21.15.123:0/1531353224 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff1700046c0 con 0x7ff184104ad0 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 0x565136dc6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 0x565136dcc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dcc700 0x565136dceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136d2bc90 con 0x565136dcc700 2024-03-24T00:37:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136cd4980 con 0x565136dc7590 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136d27d30 con 0x565136dc6ba0 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 0x565136dcc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff183400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dcc700 0x565136dceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 0x565136dcc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52048/0 (socket says 172.21.15.123:52048) 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff183e00640 1 -- 172.21.15.123:0/2774609081 learned_addr learned my addr 172.21.15.123:0/2774609081 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.929+0000 7ff188e00640 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 0x565136dc6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 -- 172.21.15.123:0/2774609081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 msgr2=0x565136dcc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 0x565136dcc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 -- 172.21.15.123:0/2774609081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dcc700 msgr2=0x565136dceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dcc700 0x565136dceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 -- 172.21.15.123:0/2774609081 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565136cec510 con 0x565136dc6ba0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff188e00640 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 0x565136dc6fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff17c037040 tx=0x7ff17c0323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54bdbbc495ef31d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff180c00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 0x7ff1680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.930+0000 7ff179600640 1 -- 172.21.15.123:0/2774609081 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c043020 con 0x565136dc6ba0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff179600640 1 -- 172.21.15.123:0/2774609081 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff17c032c90 con 0x565136dc6ba0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff179600640 1 -- 172.21.15.123:0/2774609081 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c039040 con 0x565136dc6ba0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff180c00640 1 -- 172.21.15.123:0/1531353224 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff1700cee70 con 0x7ff184104ad0 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2774609081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 msgr2=0x565136dc6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 0x565136dc6fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff17c037040 tx=0x7ff17c0323d0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff183e00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 0x7ff1680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2774609081 shutdown_connections 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dcc700 0x565136dceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc7590 0x565136dcc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2774609081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dc6ba0 0x565136dc6fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.931+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2774609081 >> 172.21.15.123:0/2774609081 conn(0x565136dc2110 msgr2=0x565136dc2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.932+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2774609081 shutdown_connections 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.932+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2774609081 wait complete. 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.932+0000 7ff183e00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 0x7ff1680a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x565136dc3080 tx=0x7ff17400b660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.932+0000 7ff18dbaf740 1 -- start start 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.932+0000 7ff18dbaf740 1 Processor -- start 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136dcf650 con 0x565136dc7590 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136ce7dd0 con 0x565136dc6ba0 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136dcf7d0 con 0x565136dcc700 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff18dbaf740 10 client.0 _ll_register_callbacks cb 0x565136a3fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52050/0 (socket says 172.21.15.123:52050) 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 -- 172.21.15.123:0/2368802027 learned_addr learned my addr 172.21.15.123:0/2368802027 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff188e00640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 -- 172.21.15.123:0/2368802027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 msgr2=0x565136dd86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 -- 172.21.15.123:0/2368802027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 msgr2=0x565136dd16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.933+0000 7ff183e00640 1 -- 172.21.15.123:0/2368802027 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565136a402d0 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff183e00640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1740042b0 tx=0x7ff1740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7f66794c43f008e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff16e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17401d620 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff17401d7c0 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff17c032c90 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff17401d9c0 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff18dbaf740 10 client.24974 fetch_fsmap learned FSMap version 19 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565136b0a6b0 con 0x565136dc6ba0 2024-03-24T00:37:12.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.934+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff17c032c90 con 0x565136dc6ba0 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff17401e420 con 0x565136dc6ba0 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 10 client.24974 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 10 client.24974 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 20 client.24974 populate_metadata read hostname 'smithi123' 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136ddda60 0x565136ddca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 4 client.24974 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565136ccd520 con 0x565136ddda60 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.935+0000 7ff183400640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136ddda60 0x565136ddca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.936+0000 7ff16e000640 10 client.24974 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.936+0000 7ff183400640 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136ddda60 0x565136ddca40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2002b691a435deb1 server_cookie=2371f64a8ff9f8b1 in_seq=0 out_seq=0 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.936+0000 7ff16e000640 1 -- 172.21.15.123:0/2368802027 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565136ccd520 con 0x565136ddda60 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.936+0000 7ff16e000640 10 client.24974 handle_command_reply: tid=0 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.936+0000 7ff18dbaf740 1 client.24974 shutdown 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 msgr2=0x565136dd11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1740042b0 tx=0x7ff1740046c0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 shutdown_connections 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136ddda60 0x565136ddca40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/2368802027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd11a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 >> 172.21.15.123:0/2368802027 conn(0x565136dc2110 msgr2=0x565136dc1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff16e000640 0 client.24974 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 shutdown_connections 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 1 -- 172.21.15.123:0/2368802027 wait complete. 2024-03-24T00:37:12.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.937+0000 7ff18dbaf740 20 client.24974 trim_cache size 0 max 16384 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.955+0000 7ff18dbaf740 1 Processor -- start 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.955+0000 7ff18dbaf740 1 -- start start 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dcfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136ddd280 con 0x565136dc7590 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136ddd8e0 con 0x565136dc6ba0 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136dddf40 con 0x565136dcc700 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52052/0 (socket says 172.21.15.123:52052) 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 -- 172.21.15.123:0/679404039 learned_addr learned my addr 172.21.15.123:0/679404039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183400640 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff188e00640 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dcfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 -- 172.21.15.123:0/679404039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 msgr2=0x565136dcfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dcfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 -- 172.21.15.123:0/679404039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 msgr2=0x565136dd8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.956+0000 7ff183e00640 1 -- 172.21.15.123:0/679404039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565136dcbe50 con 0x565136dc6ba0 2024-03-24T00:37:12.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.957+0000 7ff183e00640 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff174012d20 tx=0x7ff17400be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a4afb751891e34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.957+0000 7ff178c00640 1 -- 172.21.15.123:0/679404039 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17401e620 con 0x565136dc6ba0 2024-03-24T00:37:12.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.957+0000 7ff178c00640 1 -- 172.21.15.123:0/679404039 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff174022030 con 0x565136dc6ba0 2024-03-24T00:37:12.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.957+0000 7ff178c00640 1 -- 172.21.15.123:0/679404039 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174032450 con 0x565136dc6ba0 2024-03-24T00:37:12.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- 172.21.15.123:0/679404039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 msgr2=0x565136dceaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff174012d20 tx=0x7ff17400be70 comp rx=0 tx=0).stop 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- 172.21.15.123:0/679404039 shutdown_connections 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dcfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/679404039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dceaf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- 172.21.15.123:0/679404039 >> 172.21.15.123:0/679404039 conn(0x565136dc2110 msgr2=0x565136ddca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- 172.21.15.123:0/679404039 shutdown_connections 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- 172.21.15.123:0/679404039 wait complete. 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 -- start start 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.958+0000 7ff18dbaf740 1 Processor -- start 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136ddec00 con 0x565136dc7590 2024-03-24T00:37:12.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136dde5a0 con 0x565136dc6ba0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136ddf260 con 0x565136dcc700 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff18dbaf740 10 client.0 _ll_register_callbacks cb 0x7ff174005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52056/0 (socket says 172.21.15.123:52056) 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 -- 172.21.15.123:0/1412500398 learned_addr learned my addr 172.21.15.123:0/1412500398 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff188e00640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 -- 172.21.15.123:0/1412500398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 msgr2=0x565136dd76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 -- 172.21.15.123:0/1412500398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 msgr2=0x565136dd4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.959+0000 7ff183e00640 1 -- 172.21.15.123:0/1412500398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565136dd7ce0 con 0x565136dc6ba0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff183e00640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff1740042b0 tx=0x7ff174003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ba5548c9836b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:12.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff16e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17401e420 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff17401e5c0 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff174022030 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff17404c050 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff18dbaf740 10 client.24980 fetch_fsmap learned FSMap version 19 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.960+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565136e38ef0 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff174022030 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff174032b40 con 0x565136dc6ba0 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 10 client.24980 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 10 client.24980 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 20 client.24980 populate_metadata read hostname 'smithi123' 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136e3d0f0 0x565136e390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 4 client.24980 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:12.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565136d391d0 con 0x565136e3d0f0 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.961+0000 7ff183400640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136e3d0f0 0x565136e390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff183400640 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136e3d0f0 0x565136e390c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6561137d7b0a8b03 server_cookie=57d555810e6d1afb in_seq=0 out_seq=0 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff16e000640 10 client.24980 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff16e000640 1 -- 172.21.15.123:0/1412500398 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x565136d391d0 con 0x565136e3d0f0 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff16e000640 10 client.24980 handle_command_reply: tid=0 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 client.24980 shutdown 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 msgr2=0x565136dd41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff1740042b0 tx=0x7ff174003f50 comp rx=0 tx=0).stop 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 shutdown_connections 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565136e3d0f0 0x565136e390c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565136dcc700 0x565136dd76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565136dc7590 0x565136dd4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 --2- 172.21.15.123:0/1412500398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565136dc6ba0 0x565136dd41e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff16e000640 0 client.24980 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 >> 172.21.15.123:0/1412500398 conn(0x565136dc2110 msgr2=0x565136dcfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 shutdown_connections 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 1 -- 172.21.15.123:0/1412500398 wait complete. 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.962+0000 7ff18dbaf740 20 client.24980 trim_cache size 0 max 16384 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 24572, 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-03-24T00:37:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 99.611913028999993, 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 19, 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 89, 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 87, 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 105.936037036 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 msgr2=0x7ff1680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 0x7ff1680a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x565136dc3080 tx=0x7ff17400b660 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 msgr2=0x7ff18419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff17000e9c0 tx=0x7ff17000ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 shutdown_connections 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff1680a2880 0x7ff1680a4d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff184107400 0x7ff1841a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184104ad0 0x7ff18419a130 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 --2- 172.21.15.123:0/1531353224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1841021a0 0x7ff184199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 >> 172.21.15.123:0/1531353224 conn(0x7ff1840fbb50 msgr2=0x7ff184101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 shutdown_connections 2024-03-24T00:37:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:12.963+0000 7ff18ae00640 1 -- 172.21.15.123:0/1531353224 wait complete. 2024-03-24T00:37:12.980 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T00:37:12.981 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:37:12.991 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:37:12.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:13.564 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:13 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1767739745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:13.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:13 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1767739745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:13.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:13 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1767739745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:14 smithi204 ceph-mon[36345]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:14 smithi123 ceph-mon[31487]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:14 smithi177 ceph-mon[36034]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 -- 172.21.15.123:0/1377646776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 msgr2=0x7f261c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:15.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 --2- 172.21.15.123:0/1377646776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c104830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2610009920 tx=0x7f2610009c30 comp rx=0 tx=0).stop 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 -- 172.21.15.123:0/1377646776 shutdown_connections 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 --2- 172.21.15.123:0/1377646776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 0x7f261c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 --2- 172.21.15.123:0/1377646776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c104830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 --2- 172.21.15.123:0/1377646776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 -- 172.21.15.123:0/1377646776 >> 172.21.15.123:0/1377646776 conn(0x7f261c0ff9b0 msgr2=0x7f261c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 -- 172.21.15.123:0/1377646776 shutdown_connections 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.834+0000 7f2624000700 1 -- 172.21.15.123:0/1377646776 wait complete. 2024-03-24T00:37:15.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.835+0000 7f2624000700 1 Processor -- start 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 -- start start 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 0x7f261c1a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c110050 con 0x7f261c104410 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c10fed0 con 0x7f261c105610 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2624000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c1101d0 con 0x7f261c105fb0 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2620c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2621600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2620c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33458/0 (socket says 172.21.15.123:33458) 2024-03-24T00:37:15.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.836+0000 7f2620c00700 1 -- 172.21.15.123:0/488680491 learned_addr learned my addr 172.21.15.123:0/488680491 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2622000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 0x7f261c1a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 -- 172.21.15.123:0/488680491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 msgr2=0x7f261c1a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 0x7f261c1a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 -- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 msgr2=0x7f261c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 -- 172.21.15.123:0/488680491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c1a2320 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2620c00700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f260800ec10 tx=0x7f260800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260800cd40 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2608004500 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c1a2610 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608010690 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.837+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c1a2ba0 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.838+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c068e50 con 0x7f261c105610 2024-03-24T00:37:15.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.840+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2608003810 con 0x7f261c105610 2024-03-24T00:37:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.841+0000 7f261a000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 0x7f26040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.842+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f26080ced80 con 0x7f261c105610 2024-03-24T00:37:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.842+0000 7f2621600700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 0x7f26040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.842+0000 7f2621600700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 0x7f26040a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb70 tx=0x7f2610019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:15.845+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f260809bdd0 con 0x7f261c105610 2024-03-24T00:37:16.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.084+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f261c0625d0 con 0x7f26040a2910 2024-03-24T00:37:16.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.086+0000 7f261a000700 1 -- 172.21.15.123:0/488680491 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f261c0625d0 con 0x7f26040a2910 2024-03-24T00:37:16.087 INFO:teuthology.orchestra.run.smithi123.stdout:Backend: cephadm 2024-03-24T00:37:16.087 INFO:teuthology.orchestra.run.smithi123.stdout:Available: Yes 2024-03-24T00:37:16.087 INFO:teuthology.orchestra.run.smithi123.stdout:Paused: No 2024-03-24T00:37:16.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 msgr2=0x7f26040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:16.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 0x7f26040a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb70 tx=0x7f2610019040 comp rx=0 tx=0).stop 2024-03-24T00:37:16.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 msgr2=0x7f261c19ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:16.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f260800ec10 tx=0x7f260800ef20 comp rx=0 tx=0).stop 2024-03-24T00:37:16.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 shutdown_connections 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f26040a2910 0x7f26040a4dd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f261c105fb0 0x7f261c1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c104410 0x7f261c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 --2- 172.21.15.123:0/488680491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f261c105610 0x7f261c19ac00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 >> 172.21.15.123:0/488680491 conn(0x7f261c0ff9b0 msgr2=0x7f261c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 shutdown_connections 2024-03-24T00:37:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:16.089+0000 7f2624000700 1 -- 172.21.15.123:0/488680491 wait complete. 2024-03-24T00:37:16.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:16 smithi123 ceph-mon[31487]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:16 smithi204 ceph-mon[36345]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:16 smithi177 ceph-mon[36034]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:16.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T00:37:17.372 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:17.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:17 smithi204 ceph-mon[36345]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:17.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:17 smithi123 ceph-mon[31487]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:17 smithi177 ceph-mon[36034]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:18 smithi204 ceph-mon[36345]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:18.576 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:18 smithi123 ceph-mon[31487]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:18 smithi177 ceph-mon[36034]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:19.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 -- 172.21.15.123:0/21140512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 msgr2=0x7fc3c810c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:19.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 --2- 172.21.15.123:0/21140512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c810c520 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000b0a0 tx=0x7fc3c000b3b0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.603 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:19 smithi123 ceph-mon[31487]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 -- 172.21.15.123:0/21140512 shutdown_connections 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 --2- 172.21.15.123:0/21140512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c810c520 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 --2- 172.21.15.123:0/21140512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 0x7fc3c8074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 --2- 172.21.15.123:0/21140512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c8075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 -- 172.21.15.123:0/21140512 >> 172.21.15.123:0/21140512 conn(0x7fc3c80fd7a0 msgr2=0x7fc3c80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 -- 172.21.15.123:0/21140512 shutdown_connections 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.602+0000 7fc3d0000700 1 -- 172.21.15.123:0/21140512 wait complete. 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.603+0000 7fc3d0000700 1 Processor -- start 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 -- start start 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 0x7fc3c819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c81a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c8110010 con 0x7fc3c80761c0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c810fe90 con 0x7fc3c810a090 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3d0000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c8110190 con 0x7fc3c8074bf0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3ccc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 0x7fc3c819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3cd600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3ce000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c81a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3cd600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:36766/0 (socket says 172.21.15.123:36766) 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.604+0000 7fc3cd600700 1 -- 172.21.15.123:0/1771834535 learned_addr learned my addr 172.21.15.123:0/1771834535 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 -- 172.21.15.123:0/1771834535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 msgr2=0x7fc3c81a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c81a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 -- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 msgr2=0x7fc3c819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 0x7fc3c819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 -- 172.21.15.123:0/1771834535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c81a2250 con 0x7fc3c8074bf0 2024-03-24T00:37:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3ce000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c81a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3cd600700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9a0 tx=0x7fc3bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc004a30 con 0x7fc3c8074bf0 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c81a2540 con 0x7fc3c8074bf0 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc00ccf0 con 0x7fc3c8074bf0 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.605+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc005680 con 0x7fc3c8074bf0 2024-03-24T00:37:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.606+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c81a2a80 con 0x7fc3c8074bf0 2024-03-24T00:37:19.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.607+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c8068e50 con 0x7fc3c8074bf0 2024-03-24T00:37:19.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.609+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc3bc00c810 con 0x7fc3c8074bf0 2024-03-24T00:37:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.610+0000 7fc3c6000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 0x7fc3b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.610+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc3bc0cebf0 con 0x7fc3c8074bf0 2024-03-24T00:37:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.610+0000 7fc3ccc00700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 0x7fc3b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.611+0000 7fc3ccc00700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 0x7fc3b00a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8075ef0 tx=0x7fc3b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:19.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.614+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3bc09bc40 con 0x7fc3c8074bf0 2024-03-24T00:37:19.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:19 smithi177 ceph-mon[36034]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:19.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:19 smithi204 ceph-mon[36345]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:19.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.846+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3c80625d0 con 0x7fc3b00a2890 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.868+0000 7fc3c6000700 1 -- 172.21.15.123:0/1771834535 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fc3c80625d0 con 0x7fc3b00a2890 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.a smithi123 running (116s) 24s ago 115s 17.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e57bf069beb1 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.b smithi177 running (113s) 44s ago 112s 25.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 5f038e61ee0c 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.c smithi204 running (2m) 44s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f9a30e59b93e 2024-03-24T00:37:19.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.d smithi123 running (2m) 24s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3bf2647077ec 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.e smithi177 running (2m) 44s ago 2m 15.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e86cefa6f007 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.f smithi204 running (2m) 44s ago 2m 13.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e424664ecab2 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.g smithi123 running (119s) 24s ago 119s 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 88eb913c0207 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.h smithi177 running (2m) 44s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 629148218aec 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.i smithi204 running (2m) 44s ago 2m 16.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4181cb410092 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.j smithi123 running (2m) 24s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1658ba157df0 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.k smithi177 running (2m) 44s ago 2m 16.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 625105ea0357 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mds.l smithi204 running (2m) 44s ago 2m 13.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 0155819687c4 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.x smithi123 *:9283,8765 running (10m) 24s ago 10m 513M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 7328f1235023 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.y smithi177 *:8443,8765 running (8m) 44s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 6a1dcfd5dcba 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.z smithi204 *:8443,8765 running (8m) 44s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d16621cf933e 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mon.a smithi123 running (10m) 24s ago 10m 48.3M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1902959791ff 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mon.b smithi177 running (8m) 44s ago 8m 40.2M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 4d40ca76012f 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:mon.c smithi204 running (8m) 44s ago 8m 45.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 75b28aac96f5 2024-03-24T00:37:19.869 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (7m) 24s ago 7m 49.9M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d28b733fb979 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi177 running (7m) 44s ago 7m 50.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 173905085e36 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi204 running (7m) 44s ago 7m 51.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7739ddd5d911 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi123 running (6m) 24s ago 6m 53.4M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e28c2b670a95 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi177 running (6m) 44s ago 6m 43.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 390e1c988b66 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi204 running (5m) 44s ago 5m 46.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 39040478eb1d 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 smithi123 running (5m) 24s ago 5m 44.7M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 95cf87b14706 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 smithi177 running (4m) 44s ago 4m 47.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 9b1e261e382e 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.8 smithi204 running (4m) 44s ago 4m 46.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 53a328917d66 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.9 smithi123 running (3m) 24s ago 3m 44.3M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 be8238cb7e42 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.10 smithi177 running (3m) 44s ago 3m 54.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f58be617d030 2024-03-24T00:37:19.870 INFO:teuthology.orchestra.run.smithi123.stdout:osd.11 smithi204 running (2m) 44s ago 2m 51.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 547f00213674 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 msgr2=0x7fc3b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 0x7fc3b00a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8075ef0 tx=0x7fc3b4009260 comp rx=0 tx=0).stop 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 msgr2=0x7fc3c819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9a0 tx=0x7fc3bc00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 shutdown_connections 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc3b00a2890 0x7fc3b00a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c810a090 0x7fc3c81a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c8074bf0 0x7fc3c819a5f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 --2- 172.21.15.123:0/1771834535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3c80761c0 0x7fc3c819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:19.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 >> 172.21.15.123:0/1771834535 conn(0x7fc3c80fd7a0 msgr2=0x7fc3c8105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:19.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 shutdown_connections 2024-03-24T00:37:19.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:19.871+0000 7fc3d0000700 1 -- 172.21.15.123:0/1771834535 wait complete. 2024-03-24T00:37:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:20 smithi123 ceph-mon[31487]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:20 smithi177 ceph-mon[36034]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:20.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T00:37:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:20 smithi204 ceph-mon[36345]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:21.171 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:21.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:21 smithi123 ceph-mon[31487]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:21.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:21 smithi177 ceph-mon[36034]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:21 smithi204 ceph-mon[36345]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:23.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 -- 172.21.15.123:0/777327613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 msgr2=0x7f71ac10c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 --2- 172.21.15.123:0/777327613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac10c820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b0d0 tx=0x7f71a400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 -- 172.21.15.123:0/777327613 shutdown_connections 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 --2- 172.21.15.123:0/777327613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac10c820 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 --2- 172.21.15.123:0/777327613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1043d0 0x7f71ac1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 --2- 172.21.15.123:0/777327613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1055d0 0x7f71ac105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 -- 172.21.15.123:0/777327613 >> 172.21.15.123:0/777327613 conn(0x7f71ac0ff950 msgr2=0x7f71ac101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 -- 172.21.15.123:0/777327613 shutdown_connections 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.479+0000 7f71b3200700 1 -- 172.21.15.123:0/777327613 wait complete. 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.480+0000 7f71b3200700 1 Processor -- start 2024-03-24T00:37:23.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 -- start start 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1043d0 0x7f71ac19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1055d0 0x7f71ac19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac110010 con 0x7f71ac1055d0 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac10fe90 con 0x7f71ac105f70 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b3200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac110190 con 0x7f71ac1043d0 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71ab400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1055d0 0x7f71ac19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33484/0 (socket says 172.21.15.123:33484) 2024-03-24T00:37:23.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71b1200700 1 -- 172.21.15.123:0/1966897203 learned_addr learned my addr 172.21.15.123:0/1966897203 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:23.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.481+0000 7f71abe00700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1043d0 0x7f71ac19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 -- 172.21.15.123:0/1966897203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1043d0 msgr2=0x7f71ac19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1043d0 0x7f71ac19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 -- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1055d0 msgr2=0x7f71ac19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1055d0 0x7f71ac19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 -- 172.21.15.123:0/1966897203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac1a2200 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b1200700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b770 tx=0x7f71a40079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4003c00 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71a4003da0 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b3200700 1 -- 172.21.15.123:0/1966897203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac1a2490 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a401b9c0 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.482+0000 7f71b3200700 1 -- 172.21.15.123:0/1966897203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac1a2940 con 0x7f71ac105f70 2024-03-24T00:37:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.483+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7184005330 con 0x7f71ac105f70 2024-03-24T00:37:23.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.485+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f71a4026450 con 0x7f71ac105f70 2024-03-24T00:37:23.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.486+0000 7f71a8c00700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 0x7f71900a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.487+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f71a400f020 con 0x7f71ac105f70 2024-03-24T00:37:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.487+0000 7f71abe00700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 0x7f71900a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.487+0000 7f71abe00700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 0x7f71900a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f71ac105430 tx=0x7f719c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.491+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f71a409cd90 con 0x7f71ac105f70 2024-03-24T00:37:23.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.727+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7184000bc0 con 0x7f71900a2910 2024-03-24T00:37:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.743+0000 7f71a8c00700 1 -- 172.21.15.123:0/1966897203 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f7184000bc0 con 0x7f71900a2910 2024-03-24T00:37:23.743 INFO:teuthology.orchestra.run.smithi123.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T00:37:23.744 INFO:teuthology.orchestra.run.smithi123.stdout:mds.all 12/12 48s ago 2m smithi123=a;smithi123=d;smithi123=g;smithi123=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-03-24T00:37:23.744 INFO:teuthology.orchestra.run.smithi123.stdout:mgr 3/3 48s ago 8m smithi123=x;smithi177=y;smithi204=z;count:3 2024-03-24T00:37:23.744 INFO:teuthology.orchestra.run.smithi123.stdout:mon 3/3 48s ago 8m smithi123:172.21.15.123=a;smithi177:172.21.15.177=b;smithi204:172.21.15.204=c;count:3 2024-03-24T00:37:23.744 INFO:teuthology.orchestra.run.smithi123.stdout:osd 12 48s ago - 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.746+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 msgr2=0x7f71900a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.746+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 0x7f71900a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f71ac105430 tx=0x7f719c007400 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.746+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 msgr2=0x7f71ac1a1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.746+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b770 tx=0x7f71a40079a0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 shutdown_connections 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f71900a2910 0x7f71900a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71ac105f70 0x7f71ac1a1b60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71ac1043d0 0x7f71ac19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 --2- 172.21.15.123:0/1966897203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71ac1055d0 0x7f71ac19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 >> 172.21.15.123:0/1966897203 conn(0x7f71ac0ff950 msgr2=0x7f71ac108870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 shutdown_connections 2024-03-24T00:37:23.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:23.747+0000 7f71a2000700 1 -- 172.21.15.123:0/1966897203 wait complete. 2024-03-24T00:37:24.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:24 smithi177 ceph-mon[36034]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:24.530 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:24 smithi123 ceph-mon[31487]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:24 smithi204 ceph-mon[36345]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:24.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T00:37:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:25.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:25 smithi177 ceph-mon[36034]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:25.525 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:25 smithi123 ceph-mon[31487]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:25.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:25 smithi204 ceph-mon[36345]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:26.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:26 smithi123 ceph-mon[31487]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:26.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:26 smithi177 ceph-mon[36034]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:26.472 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:26 smithi204 ceph-mon[36345]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 -- 172.21.15.123:0/3019689940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105600 msgr2=0x7f87fc105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 --2- 172.21.15.123:0/3019689940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105600 0x7f87fc105a60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f87e80099c0 tx=0x7f87e8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 -- 172.21.15.123:0/3019689940 shutdown_connections 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 --2- 172.21.15.123:0/3019689940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105fa0 0x7f87fc10c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 --2- 172.21.15.123:0/3019689940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 0x7f87fc104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 --2- 172.21.15.123:0/3019689940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105600 0x7f87fc105a60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.187+0000 7f8802000700 1 -- 172.21.15.123:0/3019689940 >> 172.21.15.123:0/3019689940 conn(0x7f87fc0ff9a0 msgr2=0x7f87fc101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.188+0000 7f8802000700 1 -- 172.21.15.123:0/3019689940 shutdown_connections 2024-03-24T00:37:27.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.188+0000 7f8802000700 1 -- 172.21.15.123:0/3019689940 wait complete. 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.189+0000 7f8802000700 1 Processor -- start 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.189+0000 7f8802000700 1 -- start start 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.189+0000 7f8802000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 0x7f87fc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.189+0000 7f8802000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105600 0x7f87fc19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.189+0000 7f8802000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f8802000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc110040 con 0x7f87fc104400 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f8802000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc10fec0 con 0x7f87fc105fa0 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f8802000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc1101c0 con 0x7f87fc105600 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 0x7f87fc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43758/0 (socket says 172.21.15.123:43758) 2024-03-24T00:37:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 -- 172.21.15.123:0/3858070044 learned_addr learned my addr 172.21.15.123:0/3858070044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:27.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87faa00700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105600 0x7f87fc19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:27.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 -- 172.21.15.123:0/3858070044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105600 msgr2=0x7f87fc19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:27.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105600 0x7f87fc19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 -- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 msgr2=0x7f87fc19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 0x7f87fc19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.190+0000 7f87fbe00700 1 -- 172.21.15.123:0/3858070044 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc1a2280 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f87fbe00700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f400ef30 tx=0x7f87f400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f40092e0 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc1a2510 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87f4004bb0 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4004280 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.191+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc1a2a00 con 0x7f87fc105fa0 2024-03-24T00:37:27.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.192+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc068e50 con 0x7f87fc105fa0 2024-03-24T00:37:27.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.194+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f87f401b030 con 0x7f87fc105fa0 2024-03-24T00:37:27.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.195+0000 7f87f3e00700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 0x7f87e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:27.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.195+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f87f40ce520 con 0x7f87fc105fa0 2024-03-24T00:37:27.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.195+0000 7f87fb400700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 0x7f87e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:27.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.196+0000 7f87fb400700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 0x7f87e00a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f87fc105460 tx=0x7f87ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.199+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f87f4022020 con 0x7f87fc105fa0 2024-03-24T00:37:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.427+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f87fc0625d0 con 0x7f87e00a2890 2024-03-24T00:37:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.429+0000 7f87f3e00700 1 -- 172.21.15.123:0/3858070044 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f87fc0625d0 con 0x7f87e00a2890 2024-03-24T00:37:27.429 INFO:teuthology.orchestra.run.smithi123.stdout:HOST ADDR LABELS STATUS 2024-03-24T00:37:27.430 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123 172.21.15.123 2024-03-24T00:37:27.430 INFO:teuthology.orchestra.run.smithi123.stdout:smithi177 172.21.15.177 2024-03-24T00:37:27.430 INFO:teuthology.orchestra.run.smithi123.stdout:smithi204 172.21.15.204 2024-03-24T00:37:27.430 INFO:teuthology.orchestra.run.smithi123.stdout:3 hosts in cluster 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.432+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 msgr2=0x7f87e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.432+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 0x7f87e00a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f87fc105460 tx=0x7f87ec009260 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.432+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 msgr2=0x7f87fc1a1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.432+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f400ef30 tx=0x7f87f400c3a0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 shutdown_connections 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87e00a2890 0x7f87e00a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc105fa0 0x7f87fc1a1be0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87fc104400 0x7f87fc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 --2- 172.21.15.123:0/3858070044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87fc105600 0x7f87fc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 >> 172.21.15.123:0/3858070044 conn(0x7f87fc0ff9a0 msgr2=0x7f87fc108900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 shutdown_connections 2024-03-24T00:37:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:27.433+0000 7f8802000700 1 -- 172.21.15.123:0/3858070044 wait complete. 2024-03-24T00:37:28.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T00:37:28.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:28 smithi177 ceph-mon[36034]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:28 smithi204 ceph-mon[36345]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:28 smithi123 ceph-mon[31487]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:28.690 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:29.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:29 smithi177 ceph-mon[36034]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:29.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:29 smithi204 ceph-mon[36345]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:29.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:29 smithi123 ceph-mon[31487]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:30.300 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:30 smithi123 ceph-mon[31487]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:30.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:30 smithi177 ceph-mon[36034]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:30 smithi204 ceph-mon[36345]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:30.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 -- 172.21.15.123:0/2489425590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48801033c0 msgr2=0x7f4880103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 --2- 172.21.15.123:0/2489425590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48801033c0 0x7f4880103820 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f486c0099c0 tx=0x7f486c009cd0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 -- 172.21.15.123:0/2489425590 shutdown_connections 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 --2- 172.21.15.123:0/2489425590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4880103d60 0x7f488010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 --2- 172.21.15.123:0/2489425590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801021c0 0x7f48801025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 --2- 172.21.15.123:0/2489425590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48801033c0 0x7f4880103820 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 -- 172.21.15.123:0/2489425590 >> 172.21.15.123:0/2489425590 conn(0x7f48800fd7a0 msgr2=0x7f48800ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 -- 172.21.15.123:0/2489425590 shutdown_connections 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.805+0000 7f4887800700 1 -- 172.21.15.123:0/2489425590 wait complete. 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 Processor -- start 2024-03-24T00:37:30.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 -- start start 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48801021c0 0x7f488019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801033c0 0x7f488019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488010dd80 con 0x7f48801033c0 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488010dc00 con 0x7f4880103d60 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.806+0000 7f4887800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488010df00 con 0x7f48801021c0 2024-03-24T00:37:30.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f487fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801033c0 0x7f488019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43786/0 (socket says 172.21.15.123:43786) 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 -- 172.21.15.123:0/1553217875 learned_addr learned my addr 172.21.15.123:0/1553217875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4884e00700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48801021c0 0x7f488019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 -- 172.21.15.123:0/1553217875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48801021c0 msgr2=0x7f488019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48801021c0 0x7f488019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 -- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801033c0 msgr2=0x7f488019ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801033c0 0x7f488019ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 -- 172.21.15.123:0/1553217875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48801a2270 con 0x7f4880103d60 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.807+0000 7f4885800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f487800ef30 tx=0x7f487800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.808+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48780092e0 con 0x7f4880103d60 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.808+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4878004bb0 con 0x7f4880103d60 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.808+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48801a2500 con 0x7f4880103d60 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.808+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4878004280 con 0x7f4880103d60 2024-03-24T00:37:30.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.808+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48801a29f0 con 0x7f4880103d60 2024-03-24T00:37:30.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.809+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4880068e50 con 0x7f4880103d60 2024-03-24T00:37:30.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.811+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f487801b030 con 0x7f4880103d60 2024-03-24T00:37:30.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.812+0000 7f487d600700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 0x7f48680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:30.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.812+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f48780ce560 con 0x7f4880103d60 2024-03-24T00:37:30.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.812+0000 7f4884e00700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 0x7f48680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:30.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.813+0000 7f4884e00700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 0x7f48680a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4880103220 tx=0x7f4874009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:30.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:30.820+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4878022020 con 0x7f4880103d60 2024-03-24T00:37:31.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.053+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4880075ae0 con 0x7f48680a2890 2024-03-24T00:37:31.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.057+0000 7f487d600700 1 -- 172.21.15.123:0/1553217875 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f4880075ae0 con 0x7f48680a2890 2024-03-24T00:37:31.057 INFO:teuthology.orchestra.run.smithi123.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T00:37:31.057 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400YK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:37:31.057 INFO:teuthology.orchestra.run.smithi123.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:37:31.057 INFO:teuthology.orchestra.run.smithi123.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.059+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 msgr2=0x7f48680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.059+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 0x7f48680a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4880103220 tx=0x7f4874009260 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.059+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 msgr2=0x7f48801a1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.059+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f487800ef30 tx=0x7f487800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 shutdown_connections 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f48680a2890 0x7f48680a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4880103d60 0x7f48801a1bd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48801021c0 0x7f488019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 --2- 172.21.15.123:0/1553217875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48801033c0 0x7f488019ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 >> 172.21.15.123:0/1553217875 conn(0x7f48800fd7a0 msgr2=0x7f48800743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 shutdown_connections 2024-03-24T00:37:31.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:31.060+0000 7f4887800700 1 -- 172.21.15.123:0/1553217875 wait complete. 2024-03-24T00:37:31.878 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:37:31.888 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:37:31.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T00:37:32.330 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:32.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:32 smithi177 ceph-mon[36034]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:32.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:32 smithi177 ceph-mon[36034]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:32.476 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:32 smithi123 ceph-mon[31487]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:32.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:32 smithi123 ceph-mon[31487]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:32 smithi204 ceph-mon[36345]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:37:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:32 smithi204 ceph-mon[36345]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:34.352 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:34 smithi123 ceph-mon[31487]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:34.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:34 smithi177 ceph-mon[36034]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:34.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 -- 172.21.15.123:0/2671909736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105610 msgr2=0x7fb06c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 --2- 172.21.15.123:0/2671909736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105610 0x7fb06c105a70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb0580099c0 tx=0x7fb058009cd0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 -- 172.21.15.123:0/2671909736 shutdown_connections 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 --2- 172.21.15.123:0/2671909736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c105fb0 0x7fb06c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 --2- 172.21.15.123:0/2671909736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c104410 0x7fb06c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 --2- 172.21.15.123:0/2671909736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105610 0x7fb06c105a70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 -- 172.21.15.123:0/2671909736 >> 172.21.15.123:0/2671909736 conn(0x7fb06c0ff9b0 msgr2=0x7fb06c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.522+0000 7fb070e00700 1 -- 172.21.15.123:0/2671909736 shutdown_connections 2024-03-24T00:37:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.523+0000 7fb070e00700 1 -- 172.21.15.123:0/2671909736 wait complete. 2024-03-24T00:37:34.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 Processor -- start 2024-03-24T00:37:34.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 -- start start 2024-03-24T00:37:34.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:34.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 0x7fb06c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105fb0 0x7fb06c1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c110030 con 0x7fb06c105610 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c10feb0 con 0x7fb06c105fb0 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb070e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c1101b0 con 0x7fb06c104410 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb069e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb069400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 0x7fb06c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb069e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49884/0 (socket says 172.21.15.123:49884) 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.524+0000 7fb069e00700 1 -- 172.21.15.123:0/1362938351 learned_addr learned my addr 172.21.15.123:0/1362938351 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:34.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb06a800700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105fb0 0x7fb06c1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 -- 172.21.15.123:0/1362938351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105fb0 msgr2=0x7fb06c1a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105fb0 0x7fb06c1a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 -- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 msgr2=0x7fb06c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 0x7fb06c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 -- 172.21.15.123:0/1362938351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c1a2240 con 0x7fb06c104410 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069400700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 0x7fb06c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:34.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb069e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00ea80 tx=0x7fb05c00ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.525+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00cbd0 con 0x7fb06c104410 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.526+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c1a2530 con 0x7fb06c104410 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.526+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb05c00cd70 con 0x7fb06c104410 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.526+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c018760 con 0x7fb06c104410 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.526+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c1a2ac0 con 0x7fb06c104410 2024-03-24T00:37:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.527+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c068e50 con 0x7fb06c104410 2024-03-24T00:37:34.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.529+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb05c01f020 con 0x7fb06c104410 2024-03-24T00:37:34.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.530+0000 7fb062a00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 0x7fb0500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:34.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.530+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb05c0ce970 con 0x7fb06c104410 2024-03-24T00:37:34.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.530+0000 7fb069400700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 0x7fb0500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:34.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.531+0000 7fb069400700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 0x7fb0500a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb058000c00 tx=0x7fb058005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:34.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.534+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb05c01e2c0 con 0x7fb06c104410 2024-03-24T00:37:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:34 smithi204 ceph-mon[36345]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-24T00:37:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.807+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb06c062da0 con 0x7fb06c104410 2024-03-24T00:37:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.808+0000 7fb062a00700 1 -- 172.21.15.123:0/1362938351 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+2964 (secure 0 0 0) 0x7fb05c09b9c0 con 0x7fb06c104410 2024-03-24T00:37:34.808 INFO:teuthology.orchestra.run.smithi123.stdout:e19 2024-03-24T00:37:34.808 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:37:34.808 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:37:34.808 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-24T00:37:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 19 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-24T00:35:32.796117+0000 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-24T00:35:36.513248+0000 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 300 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-24T00:37:34.809 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 0 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 1 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:in 0 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=24572} 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 0 2024-03-24T00:37:34.810 INFO:teuthology.orchestra.run.smithi123.stdout:qdb_cluster leader: 24572 members: 24572 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b{0:24572} state up:active seq 3 addr [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a{0:24560} state up:standby-replay seq 1 addr [v2:172.21.15.123:6840/829451039,v1:172.21.15.123:6841/829451039] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d{-1:14397} state up:standby seq 1 addr [v2:172.21.15.123:6836/2207292600,v1:172.21.15.123:6837/2207292600] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.k{-1:24449} state up:standby seq 1 addr [v2:172.21.15.177:6832/3661966001,v1:172.21.15.177:6833/3661966001] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.204:6832/812925941,v1:172.21.15.204:6833/812925941] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.j{-1:24467} state up:standby seq 1 addr [v2:172.21.15.123:6834/548064251,v1:172.21.15.123:6835/548064251] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.e{-1:24491} state up:standby seq 1 addr [v2:172.21.15.177:6834/1091560059,v1:172.21.15.177:6835/1091560059] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.f{-1:24511} state up:standby seq 1 addr [v2:172.21.15.204:6834/1788804579,v1:172.21.15.204:6835/1788804579] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c{-1:24523} state up:standby seq 1 addr [v2:172.21.15.204:6836/1145111503,v1:172.21.15.204:6837/1145111503] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.h{-1:24524} state up:standby seq 1 addr [v2:172.21.15.177:6836/3793891487,v1:172.21.15.177:6837/3793891487] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.i{-1:24544} state up:standby seq 1 addr [v2:172.21.15.204:6838/3993860651,v1:172.21.15.204:6839/3993860651] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.812 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.g{-1:24556} state up:standby seq 1 addr [v2:172.21.15.123:6838/1841887169,v1:172.21.15.123:6839/1841887169] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:37:34.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 msgr2=0x7fb0500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:34.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 0x7fb0500a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb058000c00 tx=0x7fb058005fb0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 msgr2=0x7fb06c19a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00ea80 tx=0x7fb05c00ed90 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 shutdown_connections 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb0500a2910 0x7fb0500a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb06c105fb0 0x7fb06c1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb06c104410 0x7fb06c19a5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 --2- 172.21.15.123:0/1362938351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c105610 0x7fb06c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.812+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 >> 172.21.15.123:0/1362938351 conn(0x7fb06c0ff9b0 msgr2=0x7fb06c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.813+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 shutdown_connections 2024-03-24T00:37:34.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:34.813+0000 7fb070e00700 1 -- 172.21.15.123:0/1362938351 wait complete. 2024-03-24T00:37:34.814 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:35.564 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:35 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1362938351' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:37:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:35 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1362938351' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:37:35.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T00:37:35.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:35 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1362938351' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:37:36.083 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:36.501 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:36 smithi123 ceph-mon[31487]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:36 smithi204 ceph-mon[36345]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:36 smithi177 ceph-mon[36034]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 -- 172.21.15.123:0/1568870893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c109a10 msgr2=0x7f5b6c109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 --2- 172.21.15.123:0/1568870893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c109a10 0x7f5b6c109e70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5b580099c0 tx=0x7f5b58009cd0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 -- 172.21.15.123:0/1568870893 shutdown_connections 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 --2- 172.21.15.123:0/1568870893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 --2- 172.21.15.123:0/1568870893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c108810 0x7f5b6c108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 --2- 172.21.15.123:0/1568870893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c109a10 0x7f5b6c109e70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 -- 172.21.15.123:0/1568870893 >> 172.21.15.123:0/1568870893 conn(0x7f5b6c0777e0 msgr2=0x7f5b6c079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 -- 172.21.15.123:0/1568870893 shutdown_connections 2024-03-24T00:37:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.062+0000 7f5b72400700 1 -- 172.21.15.123:0/1568870893 wait complete. 2024-03-24T00:37:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.063+0000 7f5b72400700 1 Processor -- start 2024-03-24T00:37:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.063+0000 7f5b72400700 1 -- start start 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 0x7f5b6c19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c1a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c114400 con 0x7f5b6c108810 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c114280 con 0x7f5b6c109a10 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b72400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c114580 con 0x7f5b6c10a3b0 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 0x7f5b6c19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47566/0 (socket says 172.21.15.123:47566) 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6b400700 1 -- 172.21.15.123:0/2215389837 learned_addr learned my addr 172.21.15.123:0/2215389837 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6be00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c1a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:38.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 -- 172.21.15.123:0/2215389837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 msgr2=0x7f5b6c1a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c1a5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 -- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 msgr2=0x7f5b6c19e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6aa00700 1 -- 172.21.15.123:0/2215389837 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c1a65f0 con 0x7f5b6c109a10 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6be00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c1a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.064+0000 7f5b6b400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b6aa00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 0x7f5b6c19eed0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b580098e0 tx=0x7f5b5800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58009e60 con 0x7f5b6c109a10 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b58003e30 con 0x7f5b6c109a10 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c1a6880 con 0x7f5b6c109a10 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800fe50 con 0x7f5b6c109a10 2024-03-24T00:37:38.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.065+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1a6db0 con 0x7f5b6c109a10 2024-03-24T00:37:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.066+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c068e50 con 0x7f5b6c109a10 2024-03-24T00:37:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.068+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5b5800f580 con 0x7f5b6c109a10 2024-03-24T00:37:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.069+0000 7f5b63e00700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 0x7f5b500a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.069+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5b580cf790 con 0x7f5b6c109a10 2024-03-24T00:37:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.069+0000 7f5b6b400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 0x7f5b500a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:38.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.070+0000 7f5b6b400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 0x7f5b500a4f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c005950 tx=0x7f5b5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:38.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.073+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b58013070 con 0x7f5b6c109a10 2024-03-24T00:37:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.296+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f5b6c062da0 con 0x7f5b6c109a10 2024-03-24T00:37:38.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.297+0000 7f5b63e00700 1 -- 172.21.15.123:0/2215389837 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) v1 ==== 56+0+5310 (secure 0 0 0) 0x7f5b5809c870 con 0x7f5b6c109a10 2024-03-24T00:37:38.298 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 89 2024-03-24T00:37:38.298 INFO:teuthology.orchestra.run.smithi123.stdout:fsid 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-24T00:26:34.341313+0000 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-24T00:35:42.418066+0000 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:crush_version 26 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:full_ratio 0.95 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:backfillfull_ratio 0.9 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:nearfull_ratio 0.85 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:require_min_compat_client luminous 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:min_compat_client luminous 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:require_osd_release squid 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:stretch_mode_enabled false 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:max_osd 12 2024-03-24T00:37:38.299 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6802/407676320,v1:172.21.15.123:6803/407676320] [v2:172.21.15.123:6804/407676320,v1:172.21.15.123:6805/407676320] exists,up 5bfeabab-d57d-4786-9124-62355d56818c 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1151434087,v1:172.21.15.177:6801/1151434087] [v2:172.21.15.177:6802/1151434087,v1:172.21.15.177:6803/1151434087] exists,up 97ec847a-50c8-46e1-bee7-f7ee7a409894 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/823607592,v1:172.21.15.204:6801/823607592] [v2:172.21.15.204:6802/823607592,v1:172.21.15.204:6803/823607592] exists,up 408c94aa-43dc-4ca2-82bf-e7c8973d41d7 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 up in weight 1 up_from 26 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6810/1321829693,v1:172.21.15.123:6811/1321829693] [v2:172.21.15.123:6812/1321829693,v1:172.21.15.123:6813/1321829693] exists,up c8245aa9-2f8d-48bd-bad2-0b2e706438d2 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 up in weight 1 up_from 34 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/2757515144,v1:172.21.15.177:6809/2757515144] [v2:172.21.15.177:6810/2757515144,v1:172.21.15.177:6811/2757515144] exists,up 26f71028-4c8f-4813-ba49-216b3919c913 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 up in weight 1 up_from 39 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/635738302,v1:172.21.15.204:6809/635738302] [v2:172.21.15.204:6810/635738302,v1:172.21.15.204:6811/635738302] exists,up 34571dde-a4d1-4a14-adbc-55097581ca68 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6818/3042863338,v1:172.21.15.123:6819/3042863338] [v2:172.21.15.123:6820/3042863338,v1:172.21.15.123:6821/3042863338] exists,up 2a965499-6acb-447a-be25-fe19655bf999 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 up in weight 1 up_from 52 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/2491926780,v1:172.21.15.177:6817/2491926780] [v2:172.21.15.177:6818/2491926780,v1:172.21.15.177:6819/2491926780] exists,up 9c47f489-d125-48a6-8013-9702d5bf03f6 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.8 up in weight 1 up_from 60 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/4011283541,v1:172.21.15.204:6817/4011283541] [v2:172.21.15.204:6818/4011283541,v1:172.21.15.204:6819/4011283541] exists,up e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.9 up in weight 1 up_from 68 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6826/3491687003,v1:172.21.15.123:6827/3491687003] [v2:172.21.15.123:6828/3491687003,v1:172.21.15.123:6829/3491687003] exists,up 234e47dd-50a2-4522-8559-5cbf492d88d9 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.10 up in weight 1 up_from 74 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/2089800664,v1:172.21.15.177:6825/2089800664] [v2:172.21.15.177:6826/2089800664,v1:172.21.15.177:6827/2089800664] exists,up 52997bce-ef96-4fbd-803e-2c505beda096 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/2066593187,v1:172.21.15.204:6825/2066593187] [v2:172.21.15.204:6826/2066593187,v1:172.21.15.204:6827/2066593187] exists,up 233d66e6-e30c-427c-b37e-dda971c1b299 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 2.c [2,4] 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 2.e [2,1] 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 2.f [8,11] 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 2.1d [8,11] 2024-03-24T00:37:38.300 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 3.10 [5,9] 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 3.27 [3,9] 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 3.31 [4,8] 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:pg_upmap_items 3.36 [4,9] 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/4001225558 expires 2024-03-25T00:27:40.925167+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:6801/1263190299 expires 2024-03-25T00:27:40.925167+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:6801/3441797972 expires 2024-03-25T00:27:07.030268+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:6800/3441797972 expires 2024-03-25T00:27:07.030268+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/1307084792 expires 2024-03-25T00:27:40.925167+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:6800/1263190299 expires 2024-03-25T00:27:40.925167+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/2789972175 expires 2024-03-25T00:27:40.925167+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/828976007 expires 2024-03-25T00:27:07.030268+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/1787926427 expires 2024-03-25T00:27:07.030268+0000 2024-03-24T00:37:38.301 INFO:teuthology.orchestra.run.smithi123.stdout:blocklist 172.21.15.123:0/1360125430 expires 2024-03-25T00:27:07.030268+0000 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 msgr2=0x7f5b500a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 0x7f5b500a4f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c005950 tx=0x7f5b5c009260 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 msgr2=0x7f5b6c19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 0x7f5b6c19eed0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b580098e0 tx=0x7f5b5800bf60 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 shutdown_connections 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b500a2ab0 0x7f5b500a4f70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c10a3b0 0x7f5b6c1a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c108810 0x7f5b6c19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.301+0000 7f5b72400700 1 --2- 172.21.15.123:0/2215389837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b6c109a10 0x7f5b6c19eed0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.302+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 >> 172.21.15.123:0/2215389837 conn(0x7f5b6c0777e0 msgr2=0x7f5b6c10ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.302+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 shutdown_connections 2024-03-24T00:37:38.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:38.302+0000 7f5b72400700 1 -- 172.21.15.123:0/2215389837 wait complete. 2024-03-24T00:37:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:38 smithi204 ceph-mon[36345]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:38 smithi123 ceph-mon[31487]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:38 smithi177 ceph-mon[36034]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:39.047 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T00:37:39.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:39.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.281+0000 7fe497800640 1 -- 172.21.15.123:0/3158674759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 msgr2=0x7fe4900784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.281+0000 7fe497800640 1 --2- 172.21.15.123:0/3158674759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4900784e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe488009ca0 tx=0x7fe48802c100 comp rx=0 tx=0).stop 2024-03-24T00:37:39.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 -- 172.21.15.123:0/3158674759 shutdown_connections 2024-03-24T00:37:39.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 --2- 172.21.15.123:0/3158674759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4900784e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 --2- 172.21.15.123:0/3158674759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4900772a0 0x7fe490077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 --2- 172.21.15.123:0/3158674759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe490076060 0x7fe490076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 -- 172.21.15.123:0/3158674759 >> 172.21.15.123:0/3158674759 conn(0x7fe4900fee50 msgr2=0x7fe490101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:39.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 -- 172.21.15.123:0/3158674759 shutdown_connections 2024-03-24T00:37:39.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.282+0000 7fe497800640 1 -- 172.21.15.123:0/3158674759 wait complete. 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 Processor -- start 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 -- start start 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4900772a0 0x7fe49019e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4901a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490113970 con 0x7fe490077c40 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4901137f0 con 0x7fe490076060 2024-03-24T00:37:39.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe497800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490113af0 con 0x7fe4900772a0 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe494e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe494e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44264/0 (socket says 172.21.15.123:44264) 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe494e00640 1 -- 172.21.15.123:0/917964891 learned_addr learned my addr 172.21.15.123:0/917964891 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe495800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4901a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.283+0000 7fe48fe00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4900772a0 0x7fe49019e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 -- 172.21.15.123:0/917964891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4900772a0 msgr2=0x7fe49019e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4900772a0 0x7fe49019e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 -- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 msgr2=0x7fe4901a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4901a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 -- 172.21.15.123:0/917964891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4901a5b60 con 0x7fe490076060 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe495800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4901a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe494e00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe48400d950 tx=0x7fe48400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484017070 con 0x7fe490076060 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe484009c30 con 0x7fe490076060 2024-03-24T00:37:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48400f6b0 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4901a5e50 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.284+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4901a62b0 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.285+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490110390 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.287+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe4840109d0 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.288+0000 7fe48d600640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 0x7fe4780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.288+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe4840ce650 con 0x7fe490076060 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.288+0000 7fe48fe00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 0x7fe4780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.289+0000 7fe48fe00640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 0x7fe4780a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe49019f450 tx=0x7fe47c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:39.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.297+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe484009de0 con 0x7fe490076060 2024-03-24T00:37:39.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.499+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe490062340 con 0x7fe490076060 2024-03-24T00:37:39.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.501+0000 7fe48d600640 1 -- 172.21.15.123:0/917964891 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7fe48409b0b0 con 0x7fe490076060 2024-03-24T00:37:39.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:39.503 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:39.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 msgr2=0x7fe4780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 0x7fe4780a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe49019f450 tx=0x7fe47c005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 msgr2=0x7fe49019df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe48400d950 tx=0x7fe48400de10 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 shutdown_connections 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe4780a2800 0x7fe4780a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe490077c40 0x7fe4901a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4900772a0 0x7fe49019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.503+0000 7fe497800640 1 --2- 172.21.15.123:0/917964891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe490076060 0x7fe49019df40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.504+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 >> 172.21.15.123:0/917964891 conn(0x7fe4900fee50 msgr2=0x7fe490100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.504+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 shutdown_connections 2024-03-24T00:37:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.504+0000 7fe497800640 1 -- 172.21.15.123:0/917964891 wait complete. 2024-03-24T00:37:39.520 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:39 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2215389837' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:37:39.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:39.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:39.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:39 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2215389837' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:37:39.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:39 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2215389837' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 -- 172.21.15.123:0/3281426197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a01038e0 msgr2=0x7fc4a0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 --2- 172.21.15.123:0/3281426197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a01038e0 0x7fc4a0103d00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc490009900 tx=0x7fc49002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 -- 172.21.15.123:0/3281426197 shutdown_connections 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 --2- 172.21.15.123:0/3281426197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 0x7fc4a010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 --2- 172.21.15.123:0/3281426197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a0104ad0 0x7fc4a0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 --2- 172.21.15.123:0/3281426197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a01038e0 0x7fc4a0103d00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 -- 172.21.15.123:0/3281426197 >> 172.21.15.123:0/3281426197 conn(0x7fc4a00fee50 msgr2=0x7fc4a0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:39.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 -- 172.21.15.123:0/3281426197 shutdown_connections 2024-03-24T00:37:39.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.732+0000 7fc4a6600640 1 -- 172.21.15.123:0/3281426197 wait complete. 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 Processor -- start 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 -- start start 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a01038e0 0x7fc4a0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 0x7fc4a01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010f5a0 con 0x7fc4a0105490 2024-03-24T00:37:39.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010f420 con 0x7fc4a0104ad0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.733+0000 7fc4a6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010f720 con 0x7fc4a01038e0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 0x7fc4a01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44296/0 (socket says 172.21.15.123:44296) 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 -- 172.21.15.123:0/713387481 learned_addr learned my addr 172.21.15.123:0/713387481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49f400640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a01038e0 0x7fc4a0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 -- 172.21.15.123:0/713387481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a01038e0 msgr2=0x7fc4a0199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a01038e0 0x7fc4a0199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 -- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 msgr2=0x7fc4a01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 0x7fc4a01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 -- 172.21.15.123:0/713387481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a01a1870 con 0x7fc4a0104ad0 2024-03-24T00:37:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc49ea00640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00e9f0 tx=0x7fc48c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.734+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c00cd80 con 0x7fc4a0104ad0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.735+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc48c005500 con 0x7fc4a0104ad0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.735+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c010690 con 0x7fc4a0104ad0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.735+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a01a1b60 con 0x7fc4a0104ad0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.735+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01a2030 con 0x7fc4a0104ad0 2024-03-24T00:37:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.736+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a010ac00 con 0x7fc4a0104ad0 2024-03-24T00:37:39.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.738+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc48c0046c0 con 0x7fc4a0104ad0 2024-03-24T00:37:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.739+0000 7fc497e00640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 0x7fc4840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.740+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc48c0cef30 con 0x7fc4a0104ad0 2024-03-24T00:37:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.740+0000 7fc49f400640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 0x7fc4840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:39.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.740+0000 7fc49f400640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 0x7fc4840a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc49002c170 tx=0x7fc49002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:39.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.744+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc48c09bf80 con 0x7fc4a0104ad0 2024-03-24T00:37:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.943+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4a00622a0 con 0x7fc4a0104ad0 2024-03-24T00:37:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.943+0000 7fc497e00640 1 -- 172.21.15.123:0/713387481 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc48c09b880 con 0x7fc4a0104ad0 2024-03-24T00:37:39.945 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 msgr2=0x7fc4840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 0x7fc4840a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc49002c170 tx=0x7fc49002df90 comp rx=0 tx=0).stop 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 msgr2=0x7fc4a019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00e9f0 tx=0x7fc48c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 shutdown_connections 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc4840a2800 0x7fc4840a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a0105490 0x7fc4a01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0104ad0 0x7fc4a019a1a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 --2- 172.21.15.123:0/713387481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4a01038e0 0x7fc4a0199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 >> 172.21.15.123:0/713387481 conn(0x7fc4a00fee50 msgr2=0x7fc4a0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 shutdown_connections 2024-03-24T00:37:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:39.946+0000 7fc4a6600640 1 -- 172.21.15.123:0/713387481 wait complete. 2024-03-24T00:37:39.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:40.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 -- 172.21.15.123:0/2828904599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 msgr2=0x7f3edc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/2828904599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc101c10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009900 tx=0x7f3ec402bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:40.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 -- 172.21.15.123:0/2828904599 shutdown_connections 2024-03-24T00:37:40.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/2828904599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 0x7f3edc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/2828904599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc101c10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/2828904599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 0x7f3edc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 -- 172.21.15.123:0/2828904599 >> 172.21.15.123:0/2828904599 conn(0x7f3edc0fbb30 msgr2=0x7f3edc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 -- 172.21.15.123:0/2828904599 shutdown_connections 2024-03-24T00:37:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.185+0000 7f3ee2c00640 1 -- 172.21.15.123:0/2828904599 wait complete. 2024-03-24T00:37:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.186+0000 7f3ee2c00640 1 Processor -- start 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.186+0000 7f3ee2c00640 1 -- start start 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 0x7f3edc199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 0x7f3edc1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc10c270 con 0x7f3edc1005a0 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc10c0f0 con 0x7f3edc101790 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee2c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc10c3f0 con 0x7f3edc102150 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3edb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 0x7f3edc199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3edb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44318/0 (socket says 172.21.15.123:44318) 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3edb400640 1 -- 172.21.15.123:0/3548344018 learned_addr learned my addr 172.21.15.123:0/3548344018 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.187+0000 7f3ee0c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 0x7f3edc1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 -- 172.21.15.123:0/3548344018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 msgr2=0x7f3edc1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 0x7f3edc1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 -- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 msgr2=0x7f3edc199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 0x7f3edc199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 -- 172.21.15.123:0/3548344018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc1a1860 con 0x7f3edc101790 2024-03-24T00:37:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3edb400640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005ab0 tx=0x7f3ec402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4043020 con 0x7f3edc101790 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ec40054e0 con 0x7f3edc101790 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4003ce0 con 0x7f3edc101790 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc1a1af0 con 0x7f3edc101790 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.188+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc1a1ff0 con 0x7f3edc101790 2024-03-24T00:37:40.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.190+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc1a21c0 con 0x7f3edc101790 2024-03-24T00:37:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.193+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3ec4046020 con 0x7f3edc101790 2024-03-24T00:37:40.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.194+0000 7f3ed8c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.194+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ec40f0bd0 con 0x7f3edc101790 2024-03-24T00:37:40.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.194+0000 7f3edbe00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.195+0000 7f3edbe00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3edc19b120 tx=0x7f3ecc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:40.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.198+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ec404d020 con 0x7f3edc101790 2024-03-24T00:37:40.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.396+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3edc1a2550 con 0x7f3edc101790 2024-03-24T00:37:40.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.397+0000 7f3ed8c00640 1 -- 172.21.15.123:0/3548344018 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ec40bdc80 con 0x7f3edc101790 2024-03-24T00:37:40.398 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 msgr2=0x7f3ebc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3edc19b120 tx=0x7f3ecc00b390 comp rx=0 tx=0).stop 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 msgr2=0x7f3edc19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005ab0 tx=0x7f3ec402c6c0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 shutdown_connections 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3edc102150 0x7f3edc1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3edc101790 0x7f3edc19a190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 --2- 172.21.15.123:0/3548344018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3edc1005a0 0x7f3edc199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 >> 172.21.15.123:0/3548344018 conn(0x7f3edc0fbb30 msgr2=0x7f3edc0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 shutdown_connections 2024-03-24T00:37:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.400+0000 7f3ee2c00640 1 -- 172.21.15.123:0/3548344018 wait complete. 2024-03-24T00:37:40.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:40.501 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:40 smithi123 ceph-mon[31487]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:40.502 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:40 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/917964891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:40.502 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:40 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/713387481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:40 smithi204 ceph-mon[36345]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:40 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/917964891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:40.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:40 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/713387481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 -- 172.21.15.123:0/682413708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c58102790 msgr2=0x7f1c58102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 --2- 172.21.15.123:0/682413708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c58102790 0x7f1c58102c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009900 tx=0x7f1c4002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 -- 172.21.15.123:0/682413708 shutdown_connections 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 --2- 172.21.15.123:0/682413708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58103150 0x7f1c581099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 --2- 172.21.15.123:0/682413708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c58102790 0x7f1c58102c10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 --2- 172.21.15.123:0/682413708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c581015a0 0x7f1c581019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 -- 172.21.15.123:0/682413708 >> 172.21.15.123:0/682413708 conn(0x7f1c580fcb10 msgr2=0x7f1c580fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:40.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 -- 172.21.15.123:0/682413708 shutdown_connections 2024-03-24T00:37:40.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.638+0000 7f1c57e00640 1 -- 172.21.15.123:0/682413708 wait complete. 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 Processor -- start 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 -- start start 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 0x7f1c581978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58103150 0x7f1c5819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5810d220 con 0x7f1c581015a0 2024-03-24T00:37:40.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5810d0a0 con 0x7f1c58102790 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.639+0000 7f1c57e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5810d3a0 con 0x7f1c58103150 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c56a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 0x7f1c581978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44336/0 (socket says 172.21.15.123:44336) 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 -- 172.21.15.123:0/716700717 learned_addr learned my addr 172.21.15.123:0/716700717 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c57400640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58103150 0x7f1c5819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 -- 172.21.15.123:0/716700717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58103150 msgr2=0x7f1c5819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58103150 0x7f1c5819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 -- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 msgr2=0x7f1c581978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 0x7f1c581978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 -- 172.21.15.123:0/716700717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5819f4e0 con 0x7f1c58102790 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c56a00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 0x7f1c581978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.640+0000 7f1c4e000640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009230 tx=0x7f1c4002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:40.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.641+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40041070 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.641+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5819f770 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.641+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c40004450 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.641+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40003e20 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.641+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5819fc60 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.642+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58108990 con 0x7f1c58102790 2024-03-24T00:37:40.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.644+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1c40044070 con 0x7f1c58102790 2024-03-24T00:37:40.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.645+0000 7f1c5ca00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 0x7f1c3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:40.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.645+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1c400f06a0 con 0x7f1c58102790 2024-03-24T00:37:40.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.645+0000 7f1c56a00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 0x7f1c3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:40.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.646+0000 7f1c56a00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 0x7f1c3c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c581025f0 tx=0x7f1c48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:40.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.649+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c400bd750 con 0x7f1c58102790 2024-03-24T00:37:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:40 smithi177 ceph-mon[36034]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:40 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/917964891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:40 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/713387481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:40.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.849+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1c580622a0 con 0x7f1c58102790 2024-03-24T00:37:40.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.850+0000 7f1c5ca00640 1 -- 172.21.15.123:0/716700717 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1c400bd050 con 0x7f1c58102790 2024-03-24T00:37:40.851 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:40.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 msgr2=0x7f1c3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 0x7f1c3c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c581025f0 tx=0x7f1c48005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 msgr2=0x7f1c58197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009230 tx=0x7f1c4002c650 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 shutdown_connections 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58103150 0x7f1c5819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c58102790 0x7f1c58197e00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c3c0a2800 0x7f1c3c0a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 --2- 172.21.15.123:0/716700717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c581015a0 0x7f1c581978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.852+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 >> 172.21.15.123:0/716700717 conn(0x7f1c580fcb10 msgr2=0x7f1c58105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.853+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 shutdown_connections 2024-03-24T00:37:40.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:40.853+0000 7f1c57e00640 1 -- 172.21.15.123:0/716700717 wait complete. 2024-03-24T00:37:40.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 -- 172.21.15.123:0/2265990467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 msgr2=0x7fc248100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2265990467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248100810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009900 tx=0x7fc23c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 -- 172.21.15.123:0/2265990467 shutdown_connections 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2265990467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc248101fa0 0x7fc248108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2265990467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2481015e0 0x7fc248101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2265990467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248100810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.081+0000 7fc24ea00640 1 -- 172.21.15.123:0/2265990467 >> 172.21.15.123:0/2265990467 conn(0x7fc2480fb980 msgr2=0x7fc2480fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.082+0000 7fc24ea00640 1 -- 172.21.15.123:0/2265990467 shutdown_connections 2024-03-24T00:37:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.082+0000 7fc24ea00640 1 -- 172.21.15.123:0/2265990467 wait complete. 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.082+0000 7fc24ea00640 1 Processor -- start 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 -- start start 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 0x7fc248199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc248101fa0 0x7fc2481a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24810c0d0 con 0x7fc248101fa0 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24810bf50 con 0x7fc2481003f0 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24ea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24810c250 con 0x7fc2481015e0 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44350/0 (socket says 172.21.15.123:44350) 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24d600640 1 -- 172.21.15.123:0/2952310871 learned_addr learned my addr 172.21.15.123:0/2952310871 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24e000640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc248101fa0 0x7fc2481a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.083+0000 7fc24cc00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 0x7fc248199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 -- 172.21.15.123:0/2952310871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 msgr2=0x7fc248199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 0x7fc248199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 -- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc248101fa0 msgr2=0x7fc2481a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc248101fa0 0x7fc2481a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 -- 172.21.15.123:0/2952310871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2481a16d0 con 0x7fc2481003f0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24cc00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 0x7fc248199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24d600640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc23c02c8f0 tx=0x7fc23c005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c041070 con 0x7fc2481003f0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc23c0054e0 con 0x7fc2481003f0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2481a1960 con 0x7fc2481003f0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c034440 con 0x7fc2481003f0 2024-03-24T00:37:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.084+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2481a1e70 con 0x7fc2481003f0 2024-03-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.086+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2481a2040 con 0x7fc2481003f0 2024-03-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.088+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc23c044070 con 0x7fc2481003f0 2024-03-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.089+0000 7fc246000640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 0x7fc2300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.090+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc23c0f06d0 con 0x7fc2481003f0 2024-03-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.090+0000 7fc24cc00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 0x7fc2300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.090+0000 7fc24cc00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 0x7fc2300a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2380058c0 tx=0x7fc238005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.093+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc23c0bd780 con 0x7fc2481003f0 2024-03-24T00:37:41.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.342+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2481a2210 con 0x7fc2481003f0 2024-03-24T00:37:41.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.343+0000 7fc246000640 1 -- 172.21.15.123:0/2952310871 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fc23c0bd080 con 0x7fc2481003f0 2024-03-24T00:37:41.344 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:41.344 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:41.345 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:41.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.345+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 msgr2=0x7fc2300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.345+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 0x7fc2300a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2380058c0 tx=0x7fc238005850 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.345+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 msgr2=0x7fc248199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.345+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc23c02c8f0 tx=0x7fc23c005e40 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 shutdown_connections 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2300a2880 0x7fc2300a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc248101fa0 0x7fc2481a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2481015e0 0x7fc248199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 --2- 172.21.15.123:0/2952310871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2481003f0 0x7fc248199ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 >> 172.21.15.123:0/2952310871 conn(0x7fc2480fb980 msgr2=0x7fc2480fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 shutdown_connections 2024-03-24T00:37:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.346+0000 7fc24ea00640 1 -- 172.21.15.123:0/2952310871 wait complete. 2024-03-24T00:37:41.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:41 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3548344018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:41 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/716700717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 -- 172.21.15.123:0/1179839690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888104ad0 msgr2=0x7f3888104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 --2- 172.21.15.123:0/1179839690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888104ad0 0x7f3888104f50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3870009900 tx=0x7f387002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 -- 172.21.15.123:0/1179839690 shutdown_connections 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 --2- 172.21.15.123:0/1179839690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888105490 0x7f388810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 --2- 172.21.15.123:0/1179839690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888104ad0 0x7f3888104f50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 --2- 172.21.15.123:0/1179839690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 0x7f3888103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 -- 172.21.15.123:0/1179839690 >> 172.21.15.123:0/1179839690 conn(0x7f38880fee50 msgr2=0x7f3888101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 -- 172.21.15.123:0/1179839690 shutdown_connections 2024-03-24T00:37:41.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.573+0000 7f388ee00640 1 -- 172.21.15.123:0/1179839690 wait complete. 2024-03-24T00:37:41.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 Processor -- start 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 -- start start 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 0x7f3888199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888105490 0x7f38881a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388810f590 con 0x7f3888105490 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388810f410 con 0x7f3888104ad0 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388810f710 con 0x7f38881038e0 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f387f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f387f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44368/0 (socket says 172.21.15.123:44368) 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f387f400640 1 -- 172.21.15.123:0/2744632150 learned_addr learned my addr 172.21.15.123:0/2744632150 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.575+0000 7f388ce00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888105490 0x7f38881a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f3887e00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 0x7f3888199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f387f400640 1 -- 172.21.15.123:0/2744632150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 msgr2=0x7f3888199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f387f400640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 0x7f3888199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f387f400640 1 -- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888105490 msgr2=0x7f38881a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f387f400640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888105490 0x7f38881a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.576+0000 7f387f400640 1 -- 172.21.15.123:0/2744632150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38881a1860 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f387f400640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3870000c00 tx=0x7f387002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870043020 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38700054e0 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870003c40 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38881a1af0 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.577+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38881a2040 con 0x7f3888104ad0 2024-03-24T00:37:41.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.578+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388810ac00 con 0x7f3888104ad0 2024-03-24T00:37:41.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.580+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3870046020 con 0x7f3888104ad0 2024-03-24T00:37:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.581+0000 7f3885600640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 0x7f386c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.581+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f38700f0b20 con 0x7f3888104ad0 2024-03-24T00:37:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.581+0000 7f3887e00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 0x7f386c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.582+0000 7f3887e00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 0x7f386c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f388819b120 tx=0x7f3878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:41.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.586+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f387004d020 con 0x7f3888104ad0 2024-03-24T00:37:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:41 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3548344018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:41 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/716700717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:41 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3548344018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:41 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/716700717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:41.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.787+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38880622a0 con 0x7f3888104ad0 2024-03-24T00:37:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.790+0000 7f3885600640 1 -- 172.21.15.123:0/2744632150 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f38700bdbd0 con 0x7f3888104ad0 2024-03-24T00:37:41.790 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:41.791 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:41.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 msgr2=0x7f386c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 0x7f386c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f388819b120 tx=0x7f3878005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:41.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 msgr2=0x7f388819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:41.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3870000c00 tx=0x7f387002c650 comp rx=0 tx=0).stop 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 shutdown_connections 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f386c0a2800 0x7f386c0a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3888105490 0x7f38881a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3888104ad0 0x7f388819a190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 --2- 172.21.15.123:0/2744632150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38881038e0 0x7f3888199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 >> 172.21.15.123:0/2744632150 conn(0x7f38880fee50 msgr2=0x7f3888107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 shutdown_connections 2024-03-24T00:37:41.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:41.792+0000 7f388ee00640 1 -- 172.21.15.123:0/2744632150 wait complete. 2024-03-24T00:37:41.813 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:41.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 -- 172.21.15.123:0/497625656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c104ad0 msgr2=0x7fba0c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 --2- 172.21.15.123:0/497625656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c104ad0 0x7fba0c104f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009900 tx=0x7fb9f402bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 -- 172.21.15.123:0/497625656 shutdown_connections 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 --2- 172.21.15.123:0/497625656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c105490 0x7fba0c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 --2- 172.21.15.123:0/497625656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c104ad0 0x7fba0c104f50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 --2- 172.21.15.123:0/497625656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c1038e0 0x7fba0c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.022+0000 7fba12a00640 1 -- 172.21.15.123:0/497625656 >> 172.21.15.123:0/497625656 conn(0x7fba0c0fee50 msgr2=0x7fba0c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.023+0000 7fba12a00640 1 -- 172.21.15.123:0/497625656 shutdown_connections 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.023+0000 7fba12a00640 1 -- 172.21.15.123:0/497625656 wait complete. 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 Processor -- start 2024-03-24T00:37:42.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 -- start start 2024-03-24T00:37:42.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c104ad0 0x7fba0c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c105490 0x7fba0c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10f5b0 con 0x7fba0c104ad0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10f430 con 0x7fba0c1038e0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba12a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10f730 con 0x7fba0c105490 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44392/0 (socket says 172.21.15.123:44392) 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.024+0000 7fba0be00640 1 -- 172.21.15.123:0/2755581846 learned_addr learned my addr 172.21.15.123:0/2755581846 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba03e00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c104ad0 0x7fba0c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba10a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c105490 0x7fba0c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 -- 172.21.15.123:0/2755581846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c105490 msgr2=0x7fba0c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c105490 0x7fba0c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 -- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c104ad0 msgr2=0x7fba0c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c104ad0 0x7fba0c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 -- 172.21.15.123:0/2755581846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c1a1830 con 0x7fba0c1038e0 2024-03-24T00:37:42.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba0be00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d950 tx=0x7fb9fc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc017070 con 0x7fba0c1038e0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc009c30 con 0x7fba0c1038e0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c1a1b20 con 0x7fba0c1038e0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.025+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00f600 con 0x7fba0c1038e0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.026+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c1a2070 con 0x7fba0c1038e0 2024-03-24T00:37:42.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.027+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c10ac00 con 0x7fba0c1038e0 2024-03-24T00:37:42.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.028+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb9fc01a070 con 0x7fba0c1038e0 2024-03-24T00:37:42.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.030+0000 7fba09600640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 0x7fb9f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.030+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb9fc0ce4f0 con 0x7fba0c1038e0 2024-03-24T00:37:42.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.030+0000 7fba03e00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 0x7fb9f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.030+0000 7fba03e00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 0x7fb9f00a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402c250 tx=0x7fb9f402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.034+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9fc009de0 con 0x7fba0c1038e0 2024-03-24T00:37:42.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.233+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba0c0622d0 con 0x7fba0c1038e0 2024-03-24T00:37:42.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.233+0000 7fba09600640 1 -- 172.21.15.123:0/2755581846 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9fc09b540 con 0x7fba0c1038e0 2024-03-24T00:37:42.234 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 msgr2=0x7fb9f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 0x7fb9f00a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402c250 tx=0x7fb9f402cf90 comp rx=0 tx=0).stop 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 msgr2=0x7fba0c199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d950 tx=0x7fb9fc00de10 comp rx=0 tx=0).stop 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 shutdown_connections 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba0c105490 0x7fba0c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb9f00a2880 0x7fb9f00a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0c104ad0 0x7fba0c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 --2- 172.21.15.123:0/2755581846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba0c1038e0 0x7fba0c199c20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 >> 172.21.15.123:0/2755581846 conn(0x7fba0c0fee50 msgr2=0x7fba0c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 shutdown_connections 2024-03-24T00:37:42.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.236+0000 7fba12a00640 1 -- 172.21.15.123:0/2755581846 wait complete. 2024-03-24T00:37:42.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.462+0000 7f1f48200640 1 -- 172.21.15.123:0/1795732875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f401038e0 msgr2=0x7f1f40103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.462+0000 7f1f48200640 1 --2- 172.21.15.123:0/1795732875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f401038e0 0x7f1f40103d00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1f34009900 tx=0x7f1f3402bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 -- 172.21.15.123:0/1795732875 shutdown_connections 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 --2- 172.21.15.123:0/1795732875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f40105490 0x7f1f4010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 --2- 172.21.15.123:0/1795732875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40104ad0 0x7f1f40104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 --2- 172.21.15.123:0/1795732875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f401038e0 0x7f1f40103d00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 -- 172.21.15.123:0/1795732875 >> 172.21.15.123:0/1795732875 conn(0x7f1f400fee50 msgr2=0x7f1f40101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 -- 172.21.15.123:0/1795732875 shutdown_connections 2024-03-24T00:37:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.463+0000 7f1f48200640 1 -- 172.21.15.123:0/1795732875 wait complete. 2024-03-24T00:37:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 Processor -- start 2024-03-24T00:37:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 -- start start 2024-03-24T00:37:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 0x7f1f4019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40105490 0x7f1f401a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4010f5a0 con 0x7f1f401038e0 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4010f420 con 0x7f1f40104ad0 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f48200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4010f720 con 0x7f1f40105490 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f45800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f45800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47718/0 (socket says 172.21.15.123:47718) 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f45800640 1 -- 172.21.15.123:0/38629174 learned_addr learned my addr 172.21.15.123:0/38629174 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.464+0000 7f1f44e00640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 0x7f1f4019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f46200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40105490 0x7f1f401a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 -- 172.21.15.123:0/38629174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40105490 msgr2=0x7f1f401a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40105490 0x7f1f401a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 -- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 msgr2=0x7f1f40199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 -- 172.21.15.123:0/38629174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f401a1870 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f45800640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f44e00640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 0x7f1f4019a1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e9f0 tx=0x7f1f2c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00ccc0 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f2c005020 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0049e0 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.465+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f401a1b60 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.466+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f401a2060 con 0x7f1f40104ad0 2024-03-24T00:37:42.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.467+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4010ac80 con 0x7f1f40104ad0 2024-03-24T00:37:42.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.469+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1f2c005560 con 0x7f1f40104ad0 2024-03-24T00:37:42.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.470+0000 7f1f3e000640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 0x7f1f280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.471+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1f2c0cea00 con 0x7f1f40104ad0 2024-03-24T00:37:42.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.471+0000 7f1f45800640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 0x7f1f280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.472+0000 7f1f45800640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 0x7f1f280a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1f4019b130 tx=0x7f1f3402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.472+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f2c0051d0 con 0x7f1f40104ad0 2024-03-24T00:37:42.473 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:42 smithi123 ceph-mon[31487]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:42.474 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2952310871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.474 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2744632150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.474 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:42 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2755581846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:42 smithi204 ceph-mon[36345]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2952310871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2744632150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:42 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2755581846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:42.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.671+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f4002d2c0 con 0x7f1f40104ad0 2024-03-24T00:37:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.672+0000 7f1f3e000640 1 -- 172.21.15.123:0/38629174 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f2c09ba50 con 0x7f1f40104ad0 2024-03-24T00:37:42.673 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 msgr2=0x7f1f280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 0x7f1f280a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1f4019b130 tx=0x7f1f3402df90 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 msgr2=0x7f1f4019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 0x7f1f4019a1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e9f0 tx=0x7f1f2c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 shutdown_connections 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f40105490 0x7f1f401a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f40104ad0 0x7f1f4019a1a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1f280a2880 0x7f1f280a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 --2- 172.21.15.123:0/38629174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f401038e0 0x7f1f40199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 >> 172.21.15.123:0/38629174 conn(0x7f1f400fee50 msgr2=0x7f1f40107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 shutdown_connections 2024-03-24T00:37:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.676+0000 7f1f48200640 1 -- 172.21.15.123:0/38629174 wait complete. 2024-03-24T00:37:42.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:42 smithi177 ceph-mon[36034]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2952310871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2744632150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:42 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2755581846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:42.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 -- 172.21.15.123:0/1386235041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 msgr2=0x7f09dc102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 --2- 172.21.15.123:0/1386235041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc102c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:42.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 -- 172.21.15.123:0/1386235041 shutdown_connections 2024-03-24T00:37:42.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 --2- 172.21.15.123:0/1386235041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 --2- 172.21.15.123:0/1386235041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc102c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 --2- 172.21.15.123:0/1386235041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 0x7f09dc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 -- 172.21.15.123:0/1386235041 >> 172.21.15.123:0/1386235041 conn(0x7f09dc0fcb10 msgr2=0x7f09dc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 -- 172.21.15.123:0/1386235041 shutdown_connections 2024-03-24T00:37:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.898+0000 7f09db200640 1 -- 172.21.15.123:0/1386235041 wait complete. 2024-03-24T00:37:42.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.899+0000 7f09db200640 1 Processor -- start 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.899+0000 7f09db200640 1 -- start start 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.899+0000 7f09db200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 0x7f09dc075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.899+0000 7f09db200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09db200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09db200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc10d270 con 0x7f09dc103150 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09db200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc10d0f0 con 0x7f09dc102790 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09db200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc10d3f0 con 0x7f09dc1015a0 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09da800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09da800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47736/0 (socket says 172.21.15.123:47736) 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09da800640 1 -- 172.21.15.123:0/2656001071 learned_addr learned my addr 172.21.15.123:0/2656001071 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9e00640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 0x7f09dc075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 -- 172.21.15.123:0/2656001071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 msgr2=0x7f09dc075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 0x7f09dc075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 -- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 msgr2=0x7f09dc073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.900+0000 7f09d9400640 1 -- 172.21.15.123:0/2656001071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc074860 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09d9400640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc073a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f09c8036040 tx=0x7f09c802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8043020 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09c80054e0 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8003c30 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc1a19b0 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.901+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc1a1e80 con 0x7f09dc102790 2024-03-24T00:37:42.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.902+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc1088c0 con 0x7f09dc102790 2024-03-24T00:37:42.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.904+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f09c8046020 con 0x7f09dc102790 2024-03-24T00:37:42.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.905+0000 7f09d2a00640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.905+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f09c80f09a0 con 0x7f09dc102790 2024-03-24T00:37:42.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.905+0000 7f09d9e00640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.906+0000 7f09d9e00640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 0x7f09c00a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09dc1025f0 tx=0x7f09cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:42.909+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09c804d020 con 0x7f09dc102790 2024-03-24T00:37:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.109+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09dc0622a0 con 0x7f09dc102790 2024-03-24T00:37:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.109+0000 7f09d2a00640 1 -- 172.21.15.123:0/2656001071 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09c80bd9f0 con 0x7f09dc102790 2024-03-24T00:37:43.111 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:43.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 msgr2=0x7f09c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 0x7f09c00a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09dc1025f0 tx=0x7f09cc005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:43.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 msgr2=0x7f09dc073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc073a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f09c8036040 tx=0x7f09c802c650 comp rx=0 tx=0).stop 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 shutdown_connections 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09dc103150 0x7f09dc073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc102790 0x7f09dc073a90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 --2- 172.21.15.123:0/2656001071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09dc1015a0 0x7f09dc075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.113+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 >> 172.21.15.123:0/2656001071 conn(0x7f09dc0fcb10 msgr2=0x7f09dc105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.114+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 shutdown_connections 2024-03-24T00:37:43.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.114+0000 7f09db200640 1 -- 172.21.15.123:0/2656001071 wait complete. 2024-03-24T00:37:43.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 -- 172.21.15.123:0/516184419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90801026f0 msgr2=0x7f9080102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 --2- 172.21.15.123:0/516184419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90801026f0 0x7f9080102b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f906c009900 tx=0x7f906c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 -- 172.21.15.123:0/516184419 shutdown_connections 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 --2- 172.21.15.123:0/516184419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801030b0 0x7f9080109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 --2- 172.21.15.123:0/516184419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90801026f0 0x7f9080102b70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 --2- 172.21.15.123:0/516184419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9080101500 0x7f9080101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 -- 172.21.15.123:0/516184419 >> 172.21.15.123:0/516184419 conn(0x7f90800fca70 msgr2=0x7f90800feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 -- 172.21.15.123:0/516184419 shutdown_connections 2024-03-24T00:37:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.338+0000 7f907fe00640 1 -- 172.21.15.123:0/516184419 wait complete. 2024-03-24T00:37:43.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 Processor -- start 2024-03-24T00:37:43.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 -- start start 2024-03-24T00:37:43.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801026f0 0x7f9080199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90801030b0 0x7f90801a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908010d1e0 con 0x7f90801030b0 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908010d060 con 0x7f9080101500 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908010d360 con 0x7f90801026f0 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90801030b0 0x7f90801a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.339+0000 7f907ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44454/0 (socket says 172.21.15.123:44454) 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 -- 172.21.15.123:0/1496622367 learned_addr learned my addr 172.21.15.123:0/1496622367 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907e000640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801026f0 0x7f9080199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 -- 172.21.15.123:0/1496622367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801026f0 msgr2=0x7f9080199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801026f0 0x7f9080199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 -- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90801030b0 msgr2=0x7f90801a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90801030b0 0x7f90801a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 -- 172.21.15.123:0/1496622367 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90801a15e0 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907ea00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f907000e9c0 tx=0x7f907000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907000cd80 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90700035f0 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90801a18d0 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.340+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070010690 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.341+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90801a1df0 con 0x7f9080101500 2024-03-24T00:37:43.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.342+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90801a2050 con 0x7f9080101500 2024-03-24T00:37:43.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.343+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f90700037a0 con 0x7f9080101500 2024-03-24T00:37:43.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.344+0000 7f9077400640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 0x7f90640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.345+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f90700cee60 con 0x7f9080101500 2024-03-24T00:37:43.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.345+0000 7f907e000640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 0x7f90640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.345+0000 7f907e000640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 0x7f90640a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f906c02c240 tx=0x7f906c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.348+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f907009beb0 con 0x7f9080101500 2024-03-24T00:37:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:43 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/38629174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:43 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2656001071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.587+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90801a2220 con 0x7f9080101500 2024-03-24T00:37:43.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.588+0000 7f9077400640 1 -- 172.21.15.123:0/1496622367 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f907009b7b0 con 0x7f9080101500 2024-03-24T00:37:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 msgr2=0x7f90640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 0x7f90640a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f906c02c240 tx=0x7f906c02df90 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 msgr2=0x7f90801999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f907000e9c0 tx=0x7f907000ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 shutdown_connections 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90640a2800 0x7f90640a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90801030b0 0x7f90801a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90801026f0 0x7f9080199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 --2- 172.21.15.123:0/1496622367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9080101500 0x7f90801999c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 >> 172.21.15.123:0/1496622367 conn(0x7f90800fca70 msgr2=0x7f90800738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 shutdown_connections 2024-03-24T00:37:43.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.591+0000 7f907fe00640 1 -- 172.21.15.123:0/1496622367 wait complete. 2024-03-24T00:37:43.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:43 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/38629174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:43 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2656001071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/38629174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:43 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2656001071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:43.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 -- 172.21.15.123:0/1698015594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c104ad0 msgr2=0x7f593c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 --2- 172.21.15.123:0/1698015594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c104ad0 0x7f593c104f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5924009900 tx=0x7f592402bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 -- 172.21.15.123:0/1698015594 shutdown_connections 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 --2- 172.21.15.123:0/1698015594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 --2- 172.21.15.123:0/1698015594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c104ad0 0x7f593c104f50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 --2- 172.21.15.123:0/1698015594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c1038e0 0x7f593c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 -- 172.21.15.123:0/1698015594 >> 172.21.15.123:0/1698015594 conn(0x7f593c0fee50 msgr2=0x7f593c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 -- 172.21.15.123:0/1698015594 shutdown_connections 2024-03-24T00:37:43.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.803+0000 7f5941800640 1 -- 172.21.15.123:0/1698015594 wait complete. 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.804+0000 7f5941800640 1 Processor -- start 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 -- start start 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c104ad0 0x7f593c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c10f5b0 con 0x7f593c104ad0 2024-03-24T00:37:43.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c10f430 con 0x7f593c1038e0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f5941800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c10f730 con 0x7f593c105490 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f593aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f593aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44478/0 (socket says 172.21.15.123:44478) 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f593aa00640 1 -- 172.21.15.123:0/517447802 learned_addr learned my addr 172.21.15.123:0/517447802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f593a000640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c104ad0 0x7f593c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.805+0000 7f593b400640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 -- 172.21.15.123:0/517447802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 msgr2=0x7f593c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 -- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c104ad0 msgr2=0x7f593c19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c104ad0 0x7f593c19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 -- 172.21.15.123:0/517447802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c1a1810 con 0x7f593c1038e0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593b400640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f593aa00640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e9c0 tx=0x7f592c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c00cde0 con 0x7f593c1038e0 2024-03-24T00:37:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f592c010910 con 0x7f593c1038e0 2024-03-24T00:37:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c1a1b00 con 0x7f593c1038e0 2024-03-24T00:37:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c018a00 con 0x7f593c1038e0 2024-03-24T00:37:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.806+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c1a2070 con 0x7f593c1038e0 2024-03-24T00:37:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.807+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c10ac00 con 0x7f593c1038e0 2024-03-24T00:37:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.809+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f592c01f020 con 0x7f593c1038e0 2024-03-24T00:37:43.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.811+0000 7f5933400640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 0x7f59200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.811+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f592c0cec60 con 0x7f593c1038e0 2024-03-24T00:37:43.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.811+0000 7f593a000640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 0x7f59200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.812+0000 7f593a000640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 0x7f59200a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f592402c240 tx=0x7f592402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:43.815+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f592c09bcb0 con 0x7f593c1038e0 2024-03-24T00:37:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.009+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593c062340 con 0x7f593c1038e0 2024-03-24T00:37:44.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.011+0000 7f5933400640 1 -- 172.21.15.123:0/517447802 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f592c09b5b0 con 0x7f593c1038e0 2024-03-24T00:37:44.013 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:44.013 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 msgr2=0x7f59200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 0x7f59200a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f592402c240 tx=0x7f592402df90 comp rx=0 tx=0).stop 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 msgr2=0x7f593c199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e9c0 tx=0x7f592c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 shutdown_connections 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f593c105490 0x7f593c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59200a2880 0x7f59200a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f593c104ad0 0x7f593c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 --2- 172.21.15.123:0/517447802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f593c1038e0 0x7f593c199bf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 >> 172.21.15.123:0/517447802 conn(0x7f593c0fee50 msgr2=0x7f593c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 shutdown_connections 2024-03-24T00:37:44.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.014+0000 7f5941800640 1 -- 172.21.15.123:0/517447802 wait complete. 2024-03-24T00:37:44.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:35:32.796117+0000', 'modified': '2024-03-24T00:35:36.513248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24560': {'gid': 24560, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6841/829451039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6840', 'nonce': 829451039}, {'type': 'v1', 'addr': '172.21.15.123:6841', 'nonce': 829451039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6839/2781882004', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 2781882004}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 2781882004}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24572, 'qdb_cluster': [24572]} 2024-03-24T00:37:44.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:37:44.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.224+0000 7fe6f8200640 1 -- 172.21.15.123:0/2865493292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00fff20 msgr2=0x7fe6f0102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.224+0000 7fe6f8200640 1 --2- 172.21.15.123:0/2865493292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00fff20 0x7fe6f0102350 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009ca0 tx=0x7fe6e402c100 comp rx=0 tx=0).stop 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.224+0000 7fe6f8200640 1 -- 172.21.15.123:0/2865493292 shutdown_connections 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 --2- 172.21.15.123:0/2865493292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0105240 0x7fe6f01056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 --2- 172.21.15.123:0/2865493292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f0102890 0x7fe6f0104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 --2- 172.21.15.123:0/2865493292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00fff20 0x7fe6f0102350 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 -- 172.21.15.123:0/2865493292 >> 172.21.15.123:0/2865493292 conn(0x7fe6f00fbb40 msgr2=0x7fe6f00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 -- 172.21.15.123:0/2865493292 shutdown_connections 2024-03-24T00:37:44.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 -- 172.21.15.123:0/2865493292 wait complete. 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.225+0000 7fe6f8200640 1 Processor -- start 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 -- start start 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 0x7fe6f019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 0x7fe6f01a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0113920 con 0x7fe6f0105240 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01137a0 con 0x7fe6f00fff20 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f8200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0113aa0 con 0x7fe6f0102890 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44498/0 (socket says 172.21.15.123:44498) 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f5800640 1 -- 172.21.15.123:0/176313922 learned_addr learned my addr 172.21.15.123:0/176313922 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f6200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 0x7fe6f01a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.226+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 0x7fe6f019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 -- 172.21.15.123:0/176313922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 msgr2=0x7fe6f019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 0x7fe6f019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 -- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 msgr2=0x7fe6f01a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 0x7fe6f01a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 -- 172.21.15.123:0/176313922 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f01a5be0 con 0x7fe6f00fff20 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f6200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 0x7fe6f01a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 0x7fe6f019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f5800640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40325f0 tx=0x7fe6e4035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6ee000640 1 -- 172.21.15.123:0/176313922 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4039040 con 0x7fe6f00fff20 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6ee000640 1 -- 172.21.15.123:0/176313922 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6e4044070 con 0x7fe6f00fff20 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f01a5e70 con 0x7fe6f00fff20 2024-03-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6ee000640 1 -- 172.21.15.123:0/176313922 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4004540 con 0x7fe6f00fff20 2024-03-24T00:37:44.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.227+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f01a6270 con 0x7fe6f00fff20 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.230+0000 7fe6ee000640 1 -- 172.21.15.123:0/176313922 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe6e4032890 con 0x7fe6f00fff20 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.230+0000 7fe6faf8b740 1 Processor -- start 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.230+0000 7fe6faf8b740 1 -- start start 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5f590 0x561ff6a64090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 0x561ff6a66b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff69c3c90 con 0x561ff6a5eba0 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff696c980 con 0x561ff6a64700 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff69bfd30 con 0x561ff6a5f590 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47798/0 (socket says 172.21.15.123:47798) 2024-03-24T00:37:44.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 -- 172.21.15.123:0/4067122838 learned_addr learned my addr 172.21.15.123:0/4067122838 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 0x561ff6a66b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f5800640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5f590 0x561ff6a64090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 -- 172.21.15.123:0/4067122838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5f590 msgr2=0x561ff6a64090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5f590 0x561ff6a64090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 -- 172.21.15.123:0/4067122838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 msgr2=0x561ff6a66b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 0x561ff6a66b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 -- 172.21.15.123:0/4067122838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ff6984510 con 0x561ff6a5eba0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 0x561ff6a66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.231+0000 7fe6f6200640 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800bcf0 tx=0x7fe6e800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3ed4723ae27dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6e2a00640 1 -- 172.21.15.123:0/4067122838 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8015070 con 0x561ff6a5eba0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6e2a00640 1 -- 172.21.15.123:0/4067122838 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6e8011920 con 0x561ff6a5eba0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6e2a00640 1 -- 172.21.15.123:0/4067122838 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8010960 con 0x561ff6a5eba0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6ee000640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 0x7fe6d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6f5800640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 0x7fe6d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6ee000640 1 -- 172.21.15.123:0/176313922 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe6e40f11a0 con 0x7fe6f00fff20 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6faf8b740 1 -- 172.21.15.123:0/4067122838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 msgr2=0x561ff6a5efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.232+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800bcf0 tx=0x7fe6e800f730 comp rx=0 tx=0).stop 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6f5800640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 0x7fe6d80a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000f80 tx=0x7fe6e4032b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 -- 172.21.15.123:0/4067122838 shutdown_connections 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a64700 0x561ff6a66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5f590 0x561ff6a64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/4067122838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5eba0 0x561ff6a5efc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 -- 172.21.15.123:0/4067122838 >> 172.21.15.123:0/4067122838 conn(0x561ff6a5a110 msgr2=0x561ff6a5a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 -- 172.21.15.123:0/4067122838 shutdown_connections 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 -- 172.21.15.123:0/4067122838 wait complete. 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 -- start start 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.233+0000 7fe6faf8b740 1 Processor -- start 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5eba0 0x561ff6a69180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 0x561ff6a70690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a676b0 con 0x561ff6a64700 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff697fdd0 con 0x561ff6a5f590 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a67830 con 0x561ff6a5eba0 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6faf8b740 10 client.0 _ll_register_callbacks cb 0x561ff66d7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 0x561ff6a70690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44512/0 (socket says 172.21.15.123:44512) 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 -- 172.21.15.123:0/2987894159 learned_addr learned my addr 172.21.15.123:0/2987894159 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5eba0 0x561ff6a69180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 -- 172.21.15.123:0/2987894159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5eba0 msgr2=0x561ff6a69180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5eba0 0x561ff6a69180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 -- 172.21.15.123:0/2987894159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 msgr2=0x561ff6a70690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 0x561ff6a70690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 -- 172.21.15.123:0/2987894159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ff66d82d0 con 0x561ff6a5f590 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 0x561ff6a70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc012700 tx=0x7fe6dc012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50e832cc354b8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.234+0000 7fe6d7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc011bb0 con 0x561ff6a5f590 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe6dc011d50 con 0x561ff6a5f590 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6e8011920 con 0x561ff6a5f590 2024-03-24T00:37:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6dc005020 con 0x561ff6a5f590 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 10 client.25100 fetch_fsmap learned FSMap version 19 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561ff67a26b0 con 0x561ff6a5f590 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6e8011920 con 0x561ff6a5f590 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe6dc01c470 con 0x561ff6a5f590 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 10 client.25100 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 10 client.25100 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.235+0000 7fe6faf8b740 20 client.25100 populate_metadata read hostname 'smithi123' 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6a75950 0x561ff6a749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6faf8b740 4 client.25100 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ff6965520 con 0x561ff6a75950 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6a75950 0x561ff6a749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6a75950 0x561ff6a749f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22b57db8e7cc67bb server_cookie=f31fa0dcf6796267 in_seq=0 out_seq=0 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.236+0000 7fe6d7400640 10 client.25100 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:44.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6d7400640 1 -- 172.21.15.123:0/2987894159 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ff6965520 con 0x561ff6a75950 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6d7400640 10 client.25100 handle_command_reply: tid=0 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 client.25100 shutdown 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 msgr2=0x561ff6a696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc012700 tx=0x7fe6dc012bc0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 shutdown_connections 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a64700 0x561ff6a70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5f590 0x561ff6a696c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6a75950 0x561ff6a749f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2987894159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a5eba0 0x561ff6a69180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6d7400640 0 client.25100 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 >> 172.21.15.123:0/2987894159 conn(0x561ff6a5a110 msgr2=0x561ff6a59310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 shutdown_connections 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2987894159 wait complete. 2024-03-24T00:37:44.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.237+0000 7fe6faf8b740 20 client.25100 trim_cache size 0 max 16384 2024-03-24T00:37:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.254+0000 7fe6faf8b740 1 Processor -- start 2024-03-24T00:37:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.254+0000 7fe6faf8b740 1 -- start start 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.254+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a70650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a5a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a75170 con 0x561ff6a5f590 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a757d0 con 0x561ff6a5eba0 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a75e30 con 0x561ff6a64700 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a70650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44528/0 (socket says 172.21.15.123:44528) 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 -- 172.21.15.123:0/3068833321 learned_addr learned my addr 172.21.15.123:0/3068833321 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a5a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 -- 172.21.15.123:0/3068833321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 msgr2=0x561ff6a5a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a5a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 -- 172.21.15.123:0/3068833321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 msgr2=0x561ff6a70650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a70650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.255+0000 7fe6f6200640 1 -- 172.21.15.123:0/3068833321 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ff6a5ae80 con 0x561ff6a5eba0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6f6200640 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800f770 tx=0x7fe6e800b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eccbf404d0dfd73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6e2000640 1 -- 172.21.15.123:0/3068833321 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8024040 con 0x561ff6a5eba0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6e2000640 1 -- 172.21.15.123:0/3068833321 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6e800b380 con 0x561ff6a5eba0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6e2000640 1 -- 172.21.15.123:0/3068833321 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8015040 con 0x561ff6a5eba0 2024-03-24T00:37:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 -- 172.21.15.123:0/3068833321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 msgr2=0x561ff6a54e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800f770 tx=0x7fe6e800b800 comp rx=0 tx=0).stop 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 -- 172.21.15.123:0/3068833321 shutdown_connections 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a5a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a70650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/3068833321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a54e00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 -- 172.21.15.123:0/3068833321 >> 172.21.15.123:0/3068833321 conn(0x561ff6a5a110 msgr2=0x561ff6a67c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 -- 172.21.15.123:0/3068833321 shutdown_connections 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.256+0000 7fe6faf8b740 1 -- 172.21.15.123:0/3068833321 wait complete. 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 -- start start 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 Processor -- start 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a6cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a6fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a76af0 con 0x561ff6a5f590 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a76490 con 0x561ff6a5eba0 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff6a77150 con 0x561ff6a64700 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.257+0000 7fe6faf8b740 10 client.0 _ll_register_callbacks cb 0x7fe6dc009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44530/0 (socket says 172.21.15.123:44530) 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 -- 172.21.15.123:0/2646164258 learned_addr learned my addr 172.21.15.123:0/2646164258 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a6fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a6cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 -- 172.21.15.123:0/2646164258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 msgr2=0x561ff6a6fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a6fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 -- 172.21.15.123:0/2646164258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 msgr2=0x561ff6a6cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a6cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 -- 172.21.15.123:0/2646164258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ff6a64450 con 0x561ff6a5eba0 2024-03-24T00:37:44.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6f6200640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8005fd0 tx=0x7fe6e8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e634170fb2df573f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.258+0000 7fe6d7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.260+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8015070 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.260+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe6e8005040 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.260+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6e800b380 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6e803f050 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 10 client.25106 fetch_fsmap learned FSMap version 19 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561ff6ad1230 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6e800b380 con 0x561ff6a5eba0 2024-03-24T00:37:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe6e8004be0 con 0x561ff6a5eba0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 10 client.25106 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 10 client.25106 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 20 client.25106 populate_metadata read hostname 'smithi123' 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6ad51a0 0x561ff6ad1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 4 client.25106 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ff69d11d0 con 0x561ff6ad51a0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.262+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6ad51a0 0x561ff6ad1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6f4e00640 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6ad51a0 0x561ff6ad1400 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5631159c0872b54 server_cookie=7f85a62cc0acdb0f in_seq=0 out_seq=0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6d7400640 10 client.25106 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6d7400640 1 -- 172.21.15.123:0/2646164258 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561ff69d11d0 con 0x561ff6ad51a0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6d7400640 10 client.25106 handle_command_reply: tid=0 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 client.25106 shutdown 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 msgr2=0x561ff6a6c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8005fd0 tx=0x7fe6e8005850 comp rx=0 tx=0).stop 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 shutdown_connections 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561ff6ad51a0 0x561ff6ad1400 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561ff6a64700 0x561ff6a6fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ff6a5f590 0x561ff6a6cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6d7400640 0 client.25106 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 --2- 172.21.15.123:0/2646164258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ff6a5eba0 0x561ff6a6c880 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 >> 172.21.15.123:0/2646164258 conn(0x561ff6a5a110 msgr2=0x561ff6a54e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 shutdown_connections 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 1 -- 172.21.15.123:0/2646164258 wait complete. 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.263+0000 7fe6faf8b740 20 client.25106 trim_cache size 0 max 16384 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "082c21f6-e975-11ee-95cd-87774f69a715", 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 24572, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 130.913021031, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 19, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 89, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 87, 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 137.23714667199999 2024-03-24T00:37:44.267 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 msgr2=0x7fe6d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 0x7fe6d80a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000f80 tx=0x7fe6e4032b80 comp rx=0 tx=0).stop 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 msgr2=0x7fe6f019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40325f0 tx=0x7fe6e4035d10 comp rx=0 tx=0).stop 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 shutdown_connections 2024-03-24T00:37:44.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6d80a2800 0x7fe6d80a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f0105240 0x7fe6f01a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6f0102890 0x7fe6f019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 --2- 172.21.15.123:0/176313922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f00fff20 0x7fe6f019dfc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 >> 172.21.15.123:0/176313922 conn(0x7fe6f00fbb40 msgr2=0x7fe6f0103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 shutdown_connections 2024-03-24T00:37:44.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:44.264+0000 7fe6f8200640 1 -- 172.21.15.123:0/176313922 wait complete. 2024-03-24T00:37:44.280 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T00:37:44.304 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T00:37:44.304 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-24T00:37:44.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.304 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-03-24T00:37:44.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-03-24T00:37:44.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:44 smithi123 ceph-mon[31487]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:44.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:44 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1496622367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:44 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/517447802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.449 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2024-03-24T00:37:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:37:44.454 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-24T00:37:44.454 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:37:44.454 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:44.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T00:37:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:44 smithi204 ceph-mon[36345]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:44 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1496622367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:44 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/517447802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:44 smithi177 ceph-mon[36034]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:44 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1496622367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:44 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/517447802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:44.898 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:37:46.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:46 smithi123 ceph-mon[31487]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:46 smithi204 ceph-mon[36345]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:46 smithi177 ceph-mon[36034]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 -- 172.21.15.123:0/3825268912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf0113360 msgr2=0x7f6cf0115750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 --2- 172.21.15.123:0/3825268912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf0113360 0x7f6cf0115750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800b0d0 tx=0x7f6ce800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 -- 172.21.15.123:0/3825268912 shutdown_connections 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 --2- 172.21.15.123:0/3825268912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf0113360 0x7f6cf0115750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 --2- 172.21.15.123:0/3825268912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf00743c0 0x7f6cf00747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 --2- 172.21.15.123:0/3825268912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0074ce0 0x7f6cf0112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 -- 172.21.15.123:0/3825268912 >> 172.21.15.123:0/3825268912 conn(0x7f6cf00792f0 msgr2=0x7f6cf0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 -- 172.21.15.123:0/3825268912 shutdown_connections 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.912+0000 7f6cf5400700 1 -- 172.21.15.123:0/3825268912 wait complete. 2024-03-24T00:37:46.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.913+0000 7f6cf5400700 1 Processor -- start 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.913+0000 7f6cf5400700 1 -- start start 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf00743c0 0x7f6cf019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0113360 0x7f6cf01a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0118ba0 con 0x7f6cf0113360 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0118a20 con 0x7f6cf0074ce0 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cf5400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0118d20 con 0x7f6cf00743c0 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cee800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf00743c0 0x7f6cf019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38084/0 (socket says 172.21.15.123:38084) 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 -- 172.21.15.123:0/2578638831 learned_addr learned my addr 172.21.15.123:0/2578638831 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6cef200700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0113360 0x7f6cf01a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 -- 172.21.15.123:0/2578638831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf00743c0 msgr2=0x7f6cf019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf00743c0 0x7f6cf019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.914+0000 7f6ce5e00700 1 -- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0113360 msgr2=0x7f6cf01a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce5e00700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0113360 0x7f6cf01a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce5e00700 1 -- 172.21.15.123:0/2578638831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf01a3f20 con 0x7f6cf0074ce0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce5e00700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800d980 tx=0x7f6cd800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8017030 con 0x7f6cf0074ce0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd8004500 con 0x7f6cf0074ce0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf01a4210 con 0x7f6cf0074ce0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80039e0 con 0x7f6cf0074ce0 2024-03-24T00:37:46.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.915+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf01aba90 con 0x7f6cf0074ce0 2024-03-24T00:37:46.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.916+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf0068e50 con 0x7f6cf0074ce0 2024-03-24T00:37:46.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.918+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6cd801a070 con 0x7f6cf0074ce0 2024-03-24T00:37:46.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.919+0000 7f6ce7e00700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 0x7f6cd40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.920+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6cd80ce2b0 con 0x7f6cf0074ce0 2024-03-24T00:37:46.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.920+0000 7f6cee800700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 0x7f6cd40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.920+0000 7f6cee800700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 0x7f6cd40a4dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009680 tx=0x7f6ce0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:46.924+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6cd809b300 con 0x7f6cf0074ce0 2024-03-24T00:37:47.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.195+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6cf01abd20 con 0x7f6cf0074ce0 2024-03-24T00:37:47.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.196+0000 7f6ce7e00700 1 -- 172.21.15.123:0/2578638831 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6cd809ac00 con 0x7f6cf0074ce0 2024-03-24T00:37:47.197 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.198+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 msgr2=0x7f6cd40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.198+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 0x7f6cd40a4dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009680 tx=0x7f6ce0009260 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.198+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 msgr2=0x7f6cf019f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.198+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800d980 tx=0x7f6cd800dc90 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 shutdown_connections 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6cd40a2910 0x7f6cd40a4dd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0113360 0x7f6cf01a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6cf00743c0 0x7f6cf019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 --2- 172.21.15.123:0/2578638831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf0074ce0 0x7f6cf019f450 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 >> 172.21.15.123:0/2578638831 conn(0x7f6cf00792f0 msgr2=0x7f6cf0112050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 shutdown_connections 2024-03-24T00:37:47.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:47.199+0000 7f6cf5400700 1 -- 172.21.15.123:0/2578638831 wait complete. 2024-03-24T00:37:47.565 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2578638831' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:37:47.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:47 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2578638831' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:37:47.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:47 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2578638831' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:37:47.988 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:37:47.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:47.989 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:37:48.045 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 41978sec preferred_lft 41978sec 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-03-24T00:37:48.046 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:48.047 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:37:48.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:37:48.048 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-24T00:37:48.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:48.511 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:37:48.537 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:48 smithi123 ceph-mon[31487]: pgmap v382: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:48 smithi204 ceph-mon[36345]: pgmap v382: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:48.592 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-03-24T00:37:48.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:48.596 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-03-24T00:37:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:48 smithi177 ceph-mon[36034]: pgmap v382: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:48.724 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-03-24T00:37:48.724 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:37:48.724 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:37:48.724 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:37:48.724 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:37:48.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:48.725 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-24T00:37:48.726 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-24T00:37:48.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:37:48.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:37:48.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:37:48.726 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-24T00:37:49.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:49.435 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-03-24T00:37:49.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:49.489 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-03-24T00:37:49.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:49.545 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-24T00:37:49.545 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-24T00:37:49.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:49.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T00:37:49.545 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-24T00:37:49.689 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T00:37:49.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T00:37:49.690 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-24T00:37:49.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:49.957 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-24T00:37:49.958 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-24T00:37:49.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-03-24T00:37:49.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:37:49.958 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-24T00:37:50.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:37:50.123 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:50.171 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:37:50.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:37:50.172 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:50.220 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-24T00:37:50.220 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-24T00:37:50.221 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/4b45e73b-8124-4842-a181-0a29172dc089 2024-03-24T00:37:50.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:50.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/4b45e73b-8124-4842-a181-0a29172dc089 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-24T00:37:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:50 smithi204 ceph-mon[36345]: pgmap v383: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:50.603 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:50 smithi123 ceph-mon[31487]: pgmap v383: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:50 smithi177 ceph-mon[36034]: pgmap v383: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-24T00:37:51.009 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-24T00:37:51.009 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-24T00:37:51.009 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.123:3300,172.21.15.177:3300,172.21.15.204:3300" 2024-03-24T00:37:51.009 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with new device syntax: 0@082c21f6-e975-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/4b45e73b-8124-4842-a181-0a29172dc089 2024-03-24T00:37:51.010 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.123:3300/172.21.15.177:3300/172.21.15.204:3300" will pass to kernel 2024-03-24T00:37:51.014 INFO:teuthology.orchestra.run.smithi123.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:37:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:37:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:37:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:37:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:37:51.019 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:37:51.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:51.020 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:51.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:51.118 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:37:51.172 INFO:teuthology.orchestra.run.smithi123.stdout:client15012 2024-03-24T00:37:51.173 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:37:51.173 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:51.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:51.222 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:37:51.322 INFO:teuthology.orchestra.run.smithi123.stdout:client15012 2024-03-24T00:37:51.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:51.323 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:37:51.376 INFO:teuthology.orchestra.run.smithi123.stdout:client15012 2024-03-24T00:37:51.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:51.377 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/082c21f6-e975-11ee-95cd-87774f69a715.client15012/status') 2024-03-24T00:37:51.463 INFO:teuthology.orchestra.run.smithi123.stdout:instance: client.15012 (3)192.168.0.1:0/1041336770 2024-03-24T00:37:51.464 INFO:teuthology.orchestra.run.smithi123.stdout:blocklisted: false 2024-03-24T00:37:51.464 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records in 2024-03-24T00:37:51.464 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records out 2024-03-24T00:37:51.464 INFO:teuthology.orchestra.run.smithi123.stderr:70 bytes copied, 6.6539e-05 s, 1.1 MB/s 2024-03-24T00:37:51.469 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T00:37:51.480 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-24T00:37:51.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:51.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 -- 172.21.15.123:0/3275519473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70076060 msgr2=0x7eff70076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 --2- 172.21.15.123:0/3275519473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70076060 0x7eff70076480 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff64009900 tx=0x7eff6402bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 -- 172.21.15.123:0/3275519473 shutdown_connections 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 --2- 172.21.15.123:0/3275519473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70077c40 0x7eff700784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 --2- 172.21.15.123:0/3275519473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff70077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 --2- 172.21.15.123:0/3275519473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70076060 0x7eff70076480 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 -- 172.21.15.123:0/3275519473 >> 172.21.15.123:0/3275519473 conn(0x7eff700fee50 msgr2=0x7eff70101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 -- 172.21.15.123:0/3275519473 shutdown_connections 2024-03-24T00:37:51.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.705+0000 7eff78200640 1 -- 172.21.15.123:0/3275519473 wait complete. 2024-03-24T00:37:51.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 Processor -- start 2024-03-24T00:37:51.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 -- start start 2024-03-24T00:37:51.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70076060 0x7eff7019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff7019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff701139b0 con 0x7eff700772a0 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70113830 con 0x7eff70077c40 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff78200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70113b30 con 0x7eff70076060 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff74e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff7019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff76200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff76200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38100/0 (socket says 172.21.15.123:38100) 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff76200640 1 -- 172.21.15.123:0/1564368509 learned_addr learned my addr 172.21.15.123:0/1564368509 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff74e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff7019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38688/0 (socket says 172.21.15.123:38688) 2024-03-24T00:37:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.707+0000 7eff75800640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70076060 0x7eff7019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 -- 172.21.15.123:0/1564368509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70076060 msgr2=0x7eff7019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70076060 0x7eff7019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 -- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 msgr2=0x7eff7019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff7019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 -- 172.21.15.123:0/1564368509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff701a5bd0 con 0x7eff70077c40 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff76200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff6800eb10 tx=0x7eff6800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68004b20 con 0x7eff70077c40 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff68004cc0 con 0x7eff70077c40 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68010760 con 0x7eff70077c40 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff701a5ec0 con 0x7eff70077c40 2024-03-24T00:37:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.708+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff701a6380 con 0x7eff70077c40 2024-03-24T00:37:51.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.709+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff701103a0 con 0x7eff70077c40 2024-03-24T00:37:51.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.711+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7eff68005020 con 0x7eff70077c40 2024-03-24T00:37:51.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.712+0000 7eff6e000640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 0x7eff580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.712+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7eff680cf170 con 0x7eff70077c40 2024-03-24T00:37:51.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.713+0000 7eff75800640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 0x7eff580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.713+0000 7eff75800640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 0x7eff580a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff6402c240 tx=0x7eff6402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:51.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.716+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff6809c1c0 con 0x7eff70077c40 2024-03-24T00:37:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.917+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff70062340 con 0x7eff70077c40 2024-03-24T00:37:51.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.919+0000 7eff6e000640 1 -- 172.21.15.123:0/1564368509 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7eff6809bac0 con 0x7eff70077c40 2024-03-24T00:37:51.920 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:51.920 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:51.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.921+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 msgr2=0x7eff580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.921+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 0x7eff580a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff6402c240 tx=0x7eff6402df90 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.921+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 msgr2=0x7eff701a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.921+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff6800eb10 tx=0x7eff6800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 shutdown_connections 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff70077c40 0x7eff701a5560 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7eff580a2800 0x7eff580a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff700772a0 0x7eff7019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 --2- 172.21.15.123:0/1564368509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff70076060 0x7eff7019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 >> 172.21.15.123:0/1564368509 conn(0x7eff700fee50 msgr2=0x7eff70100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 shutdown_connections 2024-03-24T00:37:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:51.922+0000 7eff78200640 1 -- 172.21.15.123:0/1564368509 wait complete. 2024-03-24T00:37:51.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:51.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 -- 172.21.15.123:0/320314320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 msgr2=0x7f9a1c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 --2- 172.21.15.123:0/320314320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c109330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009900 tx=0x7f9a0802bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 -- 172.21.15.123:0/320314320 shutdown_connections 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 --2- 172.21.15.123:0/320314320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c109870 0x7f9a1c110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 --2- 172.21.15.123:0/320314320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c109330 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 --2- 172.21.15.123:0/320314320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c107cc0 0x7f9a1c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 -- 172.21.15.123:0/320314320 >> 172.21.15.123:0/320314320 conn(0x7f9a1c076c60 msgr2=0x7f9a1c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 -- 172.21.15.123:0/320314320 shutdown_connections 2024-03-24T00:37:52.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.148+0000 7f9a23c00640 1 -- 172.21.15.123:0/320314320 wait complete. 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 Processor -- start 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 -- start start 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c107cc0 0x7f9a1c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c113990 con 0x7f9a1c109870 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c113810 con 0x7f9a1c108eb0 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.149+0000 7f9a23c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c113b10 con 0x7f9a1c107cc0 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a21c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38120/0 (socket says 172.21.15.123:38120) 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 -- 172.21.15.123:0/931258524 learned_addr learned my addr 172.21.15.123:0/931258524 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a21c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38700/0 (socket says 172.21.15.123:38700) 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a21200640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c107cc0 0x7f9a1c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 -- 172.21.15.123:0/931258524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c107cc0 msgr2=0x7f9a1c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c107cc0 0x7f9a1c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 -- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 msgr2=0x7f9a1c1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a1be00640 1 -- 172.21.15.123:0/931258524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c1a5bc0 con 0x7f9a1c108eb0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.150+0000 7f9a21c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a1be00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a0802c8f0 tx=0x7f9a0802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a08041070 con 0x7f9a1c108eb0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a08004450 con 0x7f9a1c108eb0 2024-03-24T00:37:52.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c1a5e50 con 0x7f9a1c108eb0 2024-03-24T00:37:52.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a08034440 con 0x7f9a1c108eb0 2024-03-24T00:37:52.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.151+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c1a62c0 con 0x7f9a1c108eb0 2024-03-24T00:37:52.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.152+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c10efe0 con 0x7f9a1c108eb0 2024-03-24T00:37:52.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.154+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9a08044070 con 0x7f9a1c108eb0 2024-03-24T00:37:52.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.156+0000 7f9a19600640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 0x7f9a040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.156+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9a080f04d0 con 0x7f9a1c108eb0 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.156+0000 7f9a21200640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 0x7f9a040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.157+0000 7f9a21200640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 0x7f9a040a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9a100095f0 tx=0x7f9a10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.159+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9a080bced0 con 0x7f9a1c108eb0 2024-03-24T00:37:52.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.358+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a1c062340 con 0x7f9a1c108eb0 2024-03-24T00:37:52.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.358+0000 7f9a19600640 1 -- 172.21.15.123:0/931258524 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a080c0f70 con 0x7f9a1c108eb0 2024-03-24T00:37:52.359 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:52.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 msgr2=0x7f9a040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 0x7f9a040a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9a100095f0 tx=0x7f9a10005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:52.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 msgr2=0x7f9a1c19e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a0802c8f0 tx=0x7f9a0802c660 comp rx=0 tx=0).stop 2024-03-24T00:37:52.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 shutdown_connections 2024-03-24T00:37:52.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9a040a2880 0x7f9a040a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c109870 0x7f9a1c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a1c108eb0 0x7f9a1c19e4e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 --2- 172.21.15.123:0/931258524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a1c107cc0 0x7f9a1c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 >> 172.21.15.123:0/931258524 conn(0x7f9a1c076c60 msgr2=0x7f9a1c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 shutdown_connections 2024-03-24T00:37:52.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.361+0000 7f9a23c00640 1 -- 172.21.15.123:0/931258524 wait complete. 2024-03-24T00:37:52.382 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:52 smithi204 ceph-mon[36345]: pgmap v384: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:52 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1564368509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 -- 172.21.15.123:0/800925690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 msgr2=0x7fc2e4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 --2- 172.21.15.123:0/800925690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e4104f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009900 tx=0x7fc2d402bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 -- 172.21.15.123:0/800925690 shutdown_connections 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 --2- 172.21.15.123:0/800925690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e4105490 0x7fc2e410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 --2- 172.21.15.123:0/800925690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e4104f50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 --2- 172.21.15.123:0/800925690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e41038e0 0x7fc2e4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 -- 172.21.15.123:0/800925690 >> 172.21.15.123:0/800925690 conn(0x7fc2e40fee50 msgr2=0x7fc2e4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 -- 172.21.15.123:0/800925690 shutdown_connections 2024-03-24T00:37:52.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.597+0000 7fc2ec400640 1 -- 172.21.15.123:0/800925690 wait complete. 2024-03-24T00:37:52.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.598+0000 7fc2ec400640 1 Processor -- start 2024-03-24T00:37:52.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.598+0000 7fc2ec400640 1 -- start start 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e4105490 0x7fc2e41a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e410f5b0 con 0x7fc2e41038e0 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e410f430 con 0x7fc2e4104ad0 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ec400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e410f730 con 0x7fc2e4105490 2024-03-24T00:37:52.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38728/0 (socket says 172.21.15.123:38728) 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 -- 172.21.15.123:0/1306278356 learned_addr learned my addr 172.21.15.123:0/1306278356 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9000640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2ea400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e4105490 0x7fc2e41a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 -- 172.21.15.123:0/1306278356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e4105490 msgr2=0x7fc2e41a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e4105490 0x7fc2e41a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 -- 172.21.15.123:0/1306278356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 msgr2=0x7fc2e419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.599+0000 7fc2e9a00640 1 -- 172.21.15.123:0/1306278356 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e41a1890 con 0x7fc2e41038e0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2e9a00640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800e9c0 tx=0x7fc2d800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d800ccc0 con 0x7fc2e41038e0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d80035f0 con 0x7fc2e41038e0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e41a1b80 con 0x7fc2e41038e0 2024-03-24T00:37:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d800ce60 con 0x7fc2e41038e0 2024-03-24T00:37:52.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.600+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e41a2080 con 0x7fc2e41038e0 2024-03-24T00:37:52.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.601+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e410acc0 con 0x7fc2e41038e0 2024-03-24T00:37:52.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.603+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc2d80037a0 con 0x7fc2e41038e0 2024-03-24T00:37:52.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.605+0000 7fc2e2000640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 0x7fc2cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:52 smithi123 ceph-mon[31487]: pgmap v384: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:52 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1564368509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:52.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.605+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc2d80cee80 con 0x7fc2e41038e0 2024-03-24T00:37:52.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.605+0000 7fc2e9000640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 0x7fc2cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.606+0000 7fc2e9000640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 0x7fc2cc0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc2d402c240 tx=0x7fc2d402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.609+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc2d809bed0 con 0x7fc2e41038e0 2024-03-24T00:37:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:52 smithi177 ceph-mon[36034]: pgmap v384: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:37:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:52 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1564368509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:52.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.805+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2e4062340 con 0x7fc2e41038e0 2024-03-24T00:37:52.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.806+0000 7fc2e2000640 1 -- 172.21.15.123:0/1306278356 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2d809b7d0 con 0x7fc2e41038e0 2024-03-24T00:37:52.807 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 msgr2=0x7fc2cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 0x7fc2cc0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc2d402c240 tx=0x7fc2d402df90 comp rx=0 tx=0).stop 2024-03-24T00:37:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 msgr2=0x7fc2e4199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800e9c0 tx=0x7fc2d800edc0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 shutdown_connections 2024-03-24T00:37:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2e4105490 0x7fc2e41a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2e4104ad0 0x7fc2e419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2cc0a2800 0x7fc2cc0a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 --2- 172.21.15.123:0/1306278356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2e41038e0 0x7fc2e4199c80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 >> 172.21.15.123:0/1306278356 conn(0x7fc2e40fee50 msgr2=0x7fc2e4107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 shutdown_connections 2024-03-24T00:37:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:52.809+0000 7fc2ec400640 1 -- 172.21.15.123:0/1306278356 wait complete. 2024-03-24T00:37:52.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 -- 172.21.15.123:0/3019499816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 msgr2=0x7f653c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 --2- 172.21.15.123:0/3019499816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c077700 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6524009900 tx=0x7f652402bc50 comp rx=0 tx=0).stop 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 -- 172.21.15.123:0/3019499816 shutdown_connections 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 --2- 172.21.15.123:0/3019499816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 0x7f653c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 --2- 172.21.15.123:0/3019499816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c077700 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 --2- 172.21.15.123:0/3019499816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 -- 172.21.15.123:0/3019499816 >> 172.21.15.123:0/3019499816 conn(0x7f653c0fee50 msgr2=0x7f653c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 -- 172.21.15.123:0/3019499816 shutdown_connections 2024-03-24T00:37:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.026+0000 7f6543c00640 1 -- 172.21.15.123:0/3019499816 wait complete. 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.027+0000 7f6543c00640 1 Processor -- start 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.027+0000 7f6543c00640 1 -- start start 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.027+0000 7f6543c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.027+0000 7f6543c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6543c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 0x7f653c1a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6543c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c1139b0 con 0x7f653c076060 2024-03-24T00:37:53.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6543c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c113830 con 0x7f653c0772a0 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6543c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c113b30 con 0x7f653c077c40 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38732/0 (socket says 172.21.15.123:38732) 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 -- 172.21.15.123:0/149695896 learned_addr learned my addr 172.21.15.123:0/149695896 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6533e00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 0x7f653c1a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 -- 172.21.15.123:0/149695896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 msgr2=0x7f653c1a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 0x7f653c1a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 -- 172.21.15.123:0/149695896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 msgr2=0x7f653c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6541200640 1 -- 172.21.15.123:0/149695896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c1a5c10 con 0x7f653c076060 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.028+0000 7f6533e00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:53.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f6541200640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f652c00ea60 tx=0x7f652c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c00cde0 con 0x7f653c076060 2024-03-24T00:37:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f652c0035f0 con 0x7f653c076060 2024-03-24T00:37:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c1a5f00 con 0x7f653c076060 2024-03-24T00:37:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c010670 con 0x7f653c076060 2024-03-24T00:37:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.029+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c1a6360 con 0x7f653c076060 2024-03-24T00:37:53.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.030+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c110250 con 0x7f653c076060 2024-03-24T00:37:53.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.032+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f652c0037a0 con 0x7f653c076060 2024-03-24T00:37:53.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.033+0000 7f653a000640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 0x7f65200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.033+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f652c0cede0 con 0x7f653c076060 2024-03-24T00:37:53.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.034+0000 7f6533e00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 0x7f65200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.034+0000 7f6533e00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 0x7f65200a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f652402c260 tx=0x7f6524000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.036+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f652c09bf20 con 0x7f653c076060 2024-03-24T00:37:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.235+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f653c02d300 con 0x7f653c076060 2024-03-24T00:37:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.236+0000 7f653a000640 1 -- 172.21.15.123:0/149695896 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f652c09b820 con 0x7f653c076060 2024-03-24T00:37:53.237 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 msgr2=0x7f65200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 0x7f65200a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f652402c260 tx=0x7f6524000bc0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 msgr2=0x7f653c19dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f652c00ea60 tx=0x7f652c00ef20 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 shutdown_connections 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c077c40 0x7f653c1a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0772a0 0x7f653c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2800 0x7f65200a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 --2- 172.21.15.123:0/149695896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c076060 0x7f653c19dff0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 >> 172.21.15.123:0/149695896 conn(0x7f653c0fee50 msgr2=0x7f653c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 shutdown_connections 2024-03-24T00:37:53.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.239+0000 7f6543c00640 1 -- 172.21.15.123:0/149695896 wait complete. 2024-03-24T00:37:53.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:53.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 -- 172.21.15.123:0/708351321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541052a0 msgr2=0x7f425410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 --2- 172.21.15.123:0/708351321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541052a0 0x7f425410bb30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f424c009900 tx=0x7f424c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 -- 172.21.15.123:0/708351321 shutdown_connections 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 --2- 172.21.15.123:0/708351321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541052a0 0x7f425410bb30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 --2- 172.21.15.123:0/708351321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541048e0 0x7f4254104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 --2- 172.21.15.123:0/708351321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541036f0 0x7f4254103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 -- 172.21.15.123:0/708351321 >> 172.21.15.123:0/708351321 conn(0x7f42540fec80 msgr2=0x7f42541010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.451+0000 7f4259800640 1 -- 172.21.15.123:0/708351321 shutdown_connections 2024-03-24T00:37:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.452+0000 7f4259800640 1 -- 172.21.15.123:0/708351321 wait complete. 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.452+0000 7f4259800640 1 Processor -- start 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.452+0000 7f4259800640 1 -- start start 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541036f0 0x7f42541999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541052a0 0x7f42541a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425410f3c0 con 0x7f42541036f0 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425410f240 con 0x7f42541048e0 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4259800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425410f540 con 0x7f42541052a0 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541036f0 0x7f42541999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38182/0 (socket says 172.21.15.123:38182) 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 -- 172.21.15.123:0/807189900 learned_addr learned my addr 172.21.15.123:0/807189900 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:53.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4258e00640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541052a0 0x7f42541a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 -- 172.21.15.123:0/807189900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541052a0 msgr2=0x7f42541a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541052a0 0x7f42541a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 -- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541036f0 msgr2=0x7f42541999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541036f0 0x7f42541999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.453+0000 7f4253400640 1 -- 172.21.15.123:0/807189900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42541a1600 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4253400640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f423c009d50 tx=0x7f423c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c003b70 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f423c003d10 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c010a20 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42541a18f0 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.454+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42541a1e40 con 0x7f42541048e0 2024-03-24T00:37:53.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.455+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425410a9b0 con 0x7f42541048e0 2024-03-24T00:37:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.457+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f423c003ec0 con 0x7f42541048e0 2024-03-24T00:37:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.458+0000 7f4250c00640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 0x7f42380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.459+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f423c0cf140 con 0x7f42541048e0 2024-03-24T00:37:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.459+0000 7f4253e00640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 0x7f42380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.459+0000 7f4253e00640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 0x7f42380a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4244005fd0 tx=0x7f424400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.462+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f423c09c190 con 0x7f42541048e0 2024-03-24T00:37:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/931258524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1306278356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/149695896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/931258524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1306278356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/149695896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.661+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42540622a0 con 0x7f42541048e0 2024-03-24T00:37:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.664+0000 7f4250c00640 1 -- 172.21.15.123:0/807189900 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f423c09ba90 con 0x7f42541048e0 2024-03-24T00:37:53.665 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:53.665 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 msgr2=0x7f42380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 0x7f42380a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4244005fd0 tx=0x7f424400b390 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 msgr2=0x7f4254199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f423c009d50 tx=0x7f423c00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 shutdown_connections 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42541052a0 0x7f42541a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541048e0 0x7f4254199f20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42380a2800 0x7f42380a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 --2- 172.21.15.123:0/807189900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42541036f0 0x7f42541999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 >> 172.21.15.123:0/807189900 conn(0x7f42540fec80 msgr2=0x7f4254107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 shutdown_connections 2024-03-24T00:37:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.666+0000 7f4259800640 1 -- 172.21.15.123:0/807189900 wait complete. 2024-03-24T00:37:53.685 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:53.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/931258524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1306278356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/149695896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:53.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 -- 172.21.15.123:0/1778589054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 msgr2=0x7f8750110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 --2- 172.21.15.123:0/1778589054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f8750110070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8740009930 tx=0x7f874002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 -- 172.21.15.123:0/1778589054 shutdown_connections 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 --2- 172.21.15.123:0/1778589054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f8750110070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 --2- 172.21.15.123:0/1778589054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 0x7f8750074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 --2- 172.21.15.123:0/1778589054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 0x7f8750073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 -- 172.21.15.123:0/1778589054 >> 172.21.15.123:0/1778589054 conn(0x7f87500fcc40 msgr2=0x7f87500ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 -- 172.21.15.123:0/1778589054 shutdown_connections 2024-03-24T00:37:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.898+0000 7f8756000640 1 -- 172.21.15.123:0/1778589054 wait complete. 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.899+0000 7f8756000640 1 Processor -- start 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.899+0000 7f8756000640 1 -- start start 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 0x7f875019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 0x7f875019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750113970 con 0x7f87500756e0 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87501137f0 con 0x7f875010dc80 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f8756000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750113af0 con 0x7f8750074100 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38212/0 (socket says 172.21.15.123:38212) 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 -- 172.21.15.123:0/1335208354 learned_addr learned my addr 172.21.15.123:0/1335208354 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874ea00640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 0x7f875019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874f400640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 0x7f875019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 -- 172.21.15.123:0/1335208354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 msgr2=0x7f875019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 0x7f875019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 -- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 msgr2=0x7f875019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 0x7f875019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.900+0000 7f874fe00640 1 -- 172.21.15.123:0/1335208354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87501a5be0 con 0x7f875010dc80 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f874fe00640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f874002c8e0 tx=0x7f874002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740043020 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87400054e0 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87501a5e70 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740003d10 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.901+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87501a6330 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.902+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8750103e50 con 0x7f875010dc80 2024-03-24T00:37:53.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.904+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8740046020 con 0x7f875010dc80 2024-03-24T00:37:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.905+0000 7f8747e00640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 0x7f87340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.905+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f87400f0a40 con 0x7f875010dc80 2024-03-24T00:37:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.905+0000 7f874f400640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 0x7f87340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.906+0000 7f874f400640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 0x7f87340a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8748005fd0 tx=0x7f8748009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:53.909+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f874004d020 con 0x7f875010dc80 2024-03-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.105+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8750062340 con 0x7f875010dc80 2024-03-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.105+0000 7f8747e00640 1 -- 172.21.15.123:0/1335208354 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87400bdb80 con 0x7f875010dc80 2024-03-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 msgr2=0x7f87340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 0x7f87340a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8748005fd0 tx=0x7f8748009500 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 msgr2=0x7f87501a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f874002c8e0 tx=0x7f874002c640 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 shutdown_connections 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875010dc80 0x7f87501a5570 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87340a2800 0x7f87340a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87500756e0 0x7f875019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 --2- 172.21.15.123:0/1335208354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8750074100 0x7f875019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 >> 172.21.15.123:0/1335208354 conn(0x7f87500fcc40 msgr2=0x7f8750101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 shutdown_connections 2024-03-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.108+0000 7f8756000640 1 -- 172.21.15.123:0/1335208354 wait complete. 2024-03-24T00:37:54.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.346+0000 7fa124800640 1 -- 172.21.15.123:0/46823073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 msgr2=0x7fa11c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.346+0000 7fa124800640 1 --2- 172.21.15.123:0/46823073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c077700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009900 tx=0x7fa10c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 -- 172.21.15.123:0/46823073 shutdown_connections 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 --2- 172.21.15.123:0/46823073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 0x7fa11c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 --2- 172.21.15.123:0/46823073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c077700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 --2- 172.21.15.123:0/46823073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 0x7fa11c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 -- 172.21.15.123:0/46823073 >> 172.21.15.123:0/46823073 conn(0x7fa11c0fee50 msgr2=0x7fa11c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 -- 172.21.15.123:0/46823073 shutdown_connections 2024-03-24T00:37:54.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.347+0000 7fa124800640 1 -- 172.21.15.123:0/46823073 wait complete. 2024-03-24T00:37:54.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 Processor -- start 2024-03-24T00:37:54.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 -- start start 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 0x7fa11c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 0x7fa11c1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c1139b0 con 0x7fa11c077c40 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c113830 con 0x7fa11c0772a0 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.348+0000 7fa124800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c113b30 con 0x7fa11c076060 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38232/0 (socket says 172.21.15.123:38232) 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 -- 172.21.15.123:0/3445150764 learned_addr learned my addr 172.21.15.123:0/3445150764 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa122800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 0x7fa11c1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121e00640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 0x7fa11c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 -- 172.21.15.123:0/3445150764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 msgr2=0x7fa11c19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 0x7fa11c19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 -- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 msgr2=0x7fa11c1a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 0x7fa11c1a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 -- 172.21.15.123:0/3445150764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c1a5c30 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.349+0000 7fa121400640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa10c02c8f0 tx=0x7fa10c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.350+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c041070 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.350+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa10c0044d0 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.350+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c034440 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.350+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c1a5ec0 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.350+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c1a6330 con 0x7fa11c0772a0 2024-03-24T00:37:54.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.351+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c110250 con 0x7fa11c0772a0 2024-03-24T00:37:54.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.353+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa10c044070 con 0x7fa11c0772a0 2024-03-24T00:37:54.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.354+0000 7fa116a00640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 0x7fa1040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.354+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa10c0f0560 con 0x7fa11c0772a0 2024-03-24T00:37:54.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.354+0000 7fa121e00640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 0x7fa1040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.355+0000 7fa121e00640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 0x7fa1040a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa11000bbe0 tx=0x7fa11000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.359+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa10c0bd5b0 con 0x7fa11c0772a0 2024-03-24T00:37:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.559+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa11c062340 con 0x7fa11c0772a0 2024-03-24T00:37:54.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.559+0000 7fa116a00640 1 -- 172.21.15.123:0/3445150764 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa10c0bceb0 con 0x7fa11c0772a0 2024-03-24T00:37:54.560 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 msgr2=0x7fa1040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 0x7fa1040a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa11000bbe0 tx=0x7fa11000b390 comp rx=0 tx=0).stop 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 msgr2=0x7fa11c19e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa10c02c8f0 tx=0x7fa10c005d50 comp rx=0 tx=0).stop 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 shutdown_connections 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1040a2880 0x7fa1040a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa11c077c40 0x7fa11c1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11c0772a0 0x7fa11c19e550 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 --2- 172.21.15.123:0/3445150764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c076060 0x7fa11c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 >> 172.21.15.123:0/3445150764 conn(0x7fa11c0fee50 msgr2=0x7fa11c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 shutdown_connections 2024-03-24T00:37:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.563+0000 7fa124800640 1 -- 172.21.15.123:0/3445150764 wait complete. 2024-03-24T00:37:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:54 smithi204 ceph-mon[36345]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:37:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/807189900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1335208354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:54.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:54 smithi123 ceph-mon[31487]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:37:54.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/807189900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:54.580 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1335208354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:54.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:54 smithi177 ceph-mon[36034]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:37:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/807189900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1335208354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:54.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 -- 172.21.15.123:0/1031664537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 msgr2=0x7fc24410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 --2- 172.21.15.123:0/1031664537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc24410bc00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009ca0 tx=0x7fc23c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 -- 172.21.15.123:0/1031664537 shutdown_connections 2024-03-24T00:37:54.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 --2- 172.21.15.123:0/1031664537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc24410bc00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 --2- 172.21.15.123:0/1031664537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441049b0 0x7fc244104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 --2- 172.21.15.123:0/1031664537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441037c0 0x7fc244103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 -- 172.21.15.123:0/1031664537 >> 172.21.15.123:0/1031664537 conn(0x7fc2440fed50 msgr2=0x7fc244101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.795+0000 7fc24b000640 1 -- 172.21.15.123:0/1031664537 shutdown_connections 2024-03-24T00:37:54.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.796+0000 7fc24b000640 1 -- 172.21.15.123:0/1031664537 wait complete. 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.796+0000 7fc24b000640 1 Processor -- start 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 -- start start 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441037c0 0x7fc244199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 0x7fc244199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24410f440 con 0x7fc2441049b0 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24410f2c0 con 0x7fc244105370 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24b000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24410f5c0 con 0x7fc2441037c0 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc249200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 0x7fc244199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38250/0 (socket says 172.21.15.123:38250) 2024-03-24T00:37:54.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc24a600640 1 -- 172.21.15.123:0/2885021543 learned_addr learned my addr 172.21.15.123:0/2885021543 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:54.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.797+0000 7fc249c00640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441037c0 0x7fc244199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 -- 172.21.15.123:0/2885021543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441037c0 msgr2=0x7fc244199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441037c0 0x7fc244199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 -- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 msgr2=0x7fc244199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 0x7fc244199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 -- 172.21.15.123:0/2885021543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2441a1670 con 0x7fc244105370 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc249200640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 0x7fc244199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24a600640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc23c02c630 tx=0x7fc23c032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c041020 con 0x7fc244105370 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc23c034cd0 con 0x7fc244105370 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c039040 con 0x7fc244105370 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2441a1900 con 0x7fc244105370 2024-03-24T00:37:54.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.798+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2441a1da0 con 0x7fc244105370 2024-03-24T00:37:54.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.799+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24410aae0 con 0x7fc244105370 2024-03-24T00:37:54.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.801+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc23c044020 con 0x7fc244105370 2024-03-24T00:37:54.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.803+0000 7fc242000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 0x7fc22c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.803+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc23c0f0bf0 con 0x7fc244105370 2024-03-24T00:37:54.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.803+0000 7fc249c00640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 0x7fc22c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.804+0000 7fc249c00640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 0x7fc22c0a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc244104810 tx=0x7fc238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:54.805+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc23c04b020 con 0x7fc244105370 2024-03-24T00:37:55.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.006+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc24402d200 con 0x7fc244105370 2024-03-24T00:37:55.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.006+0000 7fc242000640 1 -- 172.21.15.123:0/2885021543 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc23c0bdcd0 con 0x7fc244105370 2024-03-24T00:37:55.007 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 msgr2=0x7fc22c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 0x7fc22c0a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc244104810 tx=0x7fc238005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 msgr2=0x7fc2441a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc23c02c630 tx=0x7fc23c032ea0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 shutdown_connections 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc244105370 0x7fc2441a1000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc22c0a2880 0x7fc22c0a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2441049b0 0x7fc244199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 --2- 172.21.15.123:0/2885021543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2441037c0 0x7fc244199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 >> 172.21.15.123:0/2885021543 conn(0x7fc2440fed50 msgr2=0x7fc244107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 shutdown_connections 2024-03-24T00:37:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.009+0000 7fc24b000640 1 -- 172.21.15.123:0/2885021543 wait complete. 2024-03-24T00:37:55.029 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-24T00:37:55.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.250+0000 7f1c86400640 1 -- 172.21.15.123:0/1855905135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c80104ad0 msgr2=0x7f1c80104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.250+0000 7f1c86400640 1 --2- 172.21.15.123:0/1855905135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c80104ad0 0x7f1c80104f50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009900 tx=0x7f1c6c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 -- 172.21.15.123:0/1855905135 shutdown_connections 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 --2- 172.21.15.123:0/1855905135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c8010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 --2- 172.21.15.123:0/1855905135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c80104ad0 0x7f1c80104f50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 --2- 172.21.15.123:0/1855905135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c801038e0 0x7f1c80103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 -- 172.21.15.123:0/1855905135 >> 172.21.15.123:0/1855905135 conn(0x7f1c800fee50 msgr2=0x7f1c80101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 -- 172.21.15.123:0/1855905135 shutdown_connections 2024-03-24T00:37:55.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.251+0000 7f1c86400640 1 -- 172.21.15.123:0/1855905135 wait complete. 2024-03-24T00:37:55.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 Processor -- start 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 -- start start 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c80104ad0 0x7f1c8019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c801a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8010f5b0 con 0x7f1c80105490 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8010f430 con 0x7f1c801038e0 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.252+0000 7f1c86400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8010f730 con 0x7f1c80104ad0 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38262/0 (socket says 172.21.15.123:38262) 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 -- 172.21.15.123:0/1968420294 learned_addr learned my addr 172.21.15.123:0/1968420294 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7fe00640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c801a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 -- 172.21.15.123:0/1968420294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c80104ad0 msgr2=0x7f1c8019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c80104ad0 0x7f1c8019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 -- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 msgr2=0x7f1c801a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c801a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 -- 172.21.15.123:0/1968420294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c801a1880 con 0x7f1c801038e0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7fe00640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c801a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.253+0000 7f1c7f400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000e9c0 tx=0x7f1c7000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.254+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000cd80 con 0x7f1c801038e0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.254+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c700035f0 con 0x7f1c801038e0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.254+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70010690 con 0x7f1c801038e0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.254+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c801a1b70 con 0x7f1c801038e0 2024-03-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.254+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c801a20c0 con 0x7f1c801038e0 2024-03-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.255+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8010ac00 con 0x7f1c801038e0 2024-03-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.257+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1c700037a0 con 0x7f1c801038e0 2024-03-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.258+0000 7f1c77e00640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 0x7f1c640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.259+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1c700cefb0 con 0x7f1c801038e0 2024-03-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.259+0000 7f1c7ea00640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 0x7f1c640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.259+0000 7f1c7ea00640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 0x7f1c640a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c0057c0 tx=0x7f1c6c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:55.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.260+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c7009c000 con 0x7f1c801038e0 2024-03-24T00:37:55.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.502+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c8002d070 con 0x7f1c801038e0 2024-03-24T00:37:55.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.503+0000 7f1c77e00640 1 -- 172.21.15.123:0/1968420294 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f1c7009b900 con 0x7f1c801038e0 2024-03-24T00:37:55.504 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:55.505 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:55.505 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:55.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.506+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 msgr2=0x7f1c640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.506+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 0x7f1c640a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c0057c0 tx=0x7f1c6c000bc0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.506+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 msgr2=0x7f1c80199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.506+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000e9c0 tx=0x7f1c7000ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 shutdown_connections 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c640a2880 0x7f1c640a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c80105490 0x7f1c801a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c80104ad0 0x7f1c8019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 --2- 172.21.15.123:0/1968420294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c801038e0 0x7f1c80199c70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 >> 172.21.15.123:0/1968420294 conn(0x7f1c800fee50 msgr2=0x7f1c80107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 shutdown_connections 2024-03-24T00:37:55.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.507+0000 7f1c86400640 1 -- 172.21.15.123:0/1968420294 wait complete. 2024-03-24T00:37:55.527 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-24T00:37:55.528 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-24T00:37:55.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:37:55.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3445150764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2885021543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3445150764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2885021543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3445150764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2885021543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 -- 172.21.15.123:0/1468306925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081052a0 msgr2=0x7f5c0810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 --2- 172.21.15.123:0/1468306925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081052a0 0x7f5c0810bb30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009ca0 tx=0x7f5c0002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 -- 172.21.15.123:0/1468306925 shutdown_connections 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 --2- 172.21.15.123:0/1468306925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081052a0 0x7f5c0810bb30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 --2- 172.21.15.123:0/1468306925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081048e0 0x7f5c08104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 --2- 172.21.15.123:0/1468306925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 -- 172.21.15.123:0/1468306925 >> 172.21.15.123:0/1468306925 conn(0x7f5c080fec80 msgr2=0x7f5c081010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 -- 172.21.15.123:0/1468306925 shutdown_connections 2024-03-24T00:37:55.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.744+0000 7f5c0e800640 1 -- 172.21.15.123:0/1468306925 wait complete. 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.745+0000 7f5c0e800640 1 Processor -- start 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.745+0000 7f5c0e800640 1 -- start start 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081052a0 0x7f5c081a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0810f370 con 0x7f5c081048e0 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0810f1f0 con 0x7f5c081036f0 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0810f4f0 con 0x7f5c081052a0 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38832/0 (socket says 172.21.15.123:38832) 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 -- 172.21.15.123:0/741688430 learned_addr learned my addr 172.21.15.123:0/741688430 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:55.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0de00640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081052a0 0x7f5c081a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 -- 172.21.15.123:0/741688430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081052a0 msgr2=0x7f5c081a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081052a0 0x7f5c081a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 -- 172.21.15.123:0/741688430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 msgr2=0x7f5c08199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0ca00640 1 -- 172.21.15.123:0/741688430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c081a1570 con 0x7f5c081048e0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.746+0000 7f5c0d400640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c0ca00640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e9f0 tx=0x7f5bf400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400cd80 con 0x7f5c081048e0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c081a1860 con 0x7f5c081048e0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf4005500 con 0x7f5c081048e0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4010690 con 0x7f5c081048e0 2024-03-24T00:37:55.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.747+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c081a1d60 con 0x7f5c081048e0 2024-03-24T00:37:55.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.748+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0810a9b0 con 0x7f5c081048e0 2024-03-24T00:37:55.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.750+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5bf40046c0 con 0x7f5c081048e0 2024-03-24T00:37:55.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.751+0000 7f5c06000640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 0x7f5bf00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:55.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.752+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5bf40ceea0 con 0x7f5c081048e0 2024-03-24T00:37:55.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.752+0000 7f5c0d400640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 0x7f5bf00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:55.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.752+0000 7f5c0d400640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 0x7f5bf00a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c08104740 tx=0x7f5bfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:55.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.756+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5bf409bef0 con 0x7f5c081048e0 2024-03-24T00:37:55.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.952+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c08061ae0 con 0x7f5bf00a2800 2024-03-24T00:37:55.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.962+0000 7f5c06000640 1 -- 172.21.15.123:0/741688430 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258757 (secure 0 0 0) 0x7f5c08061ae0 con 0x7f5bf00a2800 2024-03-24T00:37:55.967 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:55.969 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.968+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 msgr2=0x7f5bf00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.968+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 0x7f5bf00a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c08104740 tx=0x7f5bfc005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.968+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 msgr2=0x7f5c08199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.968+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e9f0 tx=0x7f5bf400edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.968+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 shutdown_connections 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c081052a0 0x7f5c081a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5bf00a2800 0x7f5bf00a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c081048e0 0x7f5c08199e90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 --2- 172.21.15.123:0/741688430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c081036f0 0x7f5c08199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 >> 172.21.15.123:0/741688430 conn(0x7f5c080fec80 msgr2=0x7f5c08107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 shutdown_connections 2024-03-24T00:37:55.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:55.969+0000 7f5c0e800640 1 -- 172.21.15.123:0/741688430 wait complete. 2024-03-24T00:37:55.990 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-03-24T00:37:55.125464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659224,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":420,"num_read_kb":325,"num_write":272,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354108,"kb_used_data":13276,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587716,"statfs":{"total":1151940427776,"available":1151577821184,"internally_reserved":0,"allocated":13594624,"data_stored":7557612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5679,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004698"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082792+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:36:41.082792+0000","last_peered":"2024-03-24T00:36:41.082792+0000","last_clean":"2024-03-24T00:36:41.082792+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:36:41.082792+0000","last_undegraded":"2024-03-24T00:36:41.082792+0000","last_fullsized":"2024-03-24T00:36:41.082792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083200+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:36:41.083200+0000","last_peered":"2024-03-24T00:36:41.083200+0000","last_clean":"2024-03-24T00:36:41.083200+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:36:41.083200+0000","last_undegraded":"2024-03-24T00:36:41.083200+0000","last_fullsized":"2024-03-24T00:36:41.083200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082591+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:36:41.082591+0000","last_peered":"2024-03-24T00:36:41.082591+0000","last_clean":"2024-03-24T00:36:41.082591+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:36:41.082591+0000","last_undegraded":"2024-03-24T00:36:41.082591+0000","last_fullsized":"2024-03-24T00:36:41.082591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059003+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:36:41.059003+0000","last_peered":"2024-03-24T00:36:41.059003+0000","last_clean":"2024-03-24T00:36:41.059003+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:36:41.059003+0000","last_undegraded":"2024-03-24T00:36:41.059003+0000","last_fullsized":"2024-03-24T00:36:41.059003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.068426+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:36:41.068426+0000","last_peered":"2024-03-24T00:36:41.068426+0000","last_clean":"2024-03-24T00:36:41.068426+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:36:41.068426+0000","last_undegraded":"2024-03-24T00:36:41.068426+0000","last_fullsized":"2024-03-24T00:36:41.068426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073848+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:36:41.073848+0000","last_peered":"2024-03-24T00:36:41.073848+0000","last_clean":"2024-03-24T00:36:41.073848+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:36:41.073848+0000","last_undegraded":"2024-03-24T00:36:41.073848+0000","last_fullsized":"2024-03-24T00:36:41.073848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074753+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:36:41.074753+0000","last_peered":"2024-03-24T00:36:41.074753+0000","last_clean":"2024-03-24T00:36:41.074753+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:36:41.074753+0000","last_undegraded":"2024-03-24T00:36:41.074753+0000","last_fullsized":"2024-03-24T00:36:41.074753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065233+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:36:41.065233+0000","last_peered":"2024-03-24T00:36:41.065233+0000","last_clean":"2024-03-24T00:36:41.065233+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:36:41.065233+0000","last_undegraded":"2024-03-24T00:36:41.065233+0000","last_fullsized":"2024-03-24T00:36:41.065233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063340+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:36:41.063340+0000","last_peered":"2024-03-24T00:36:41.063340+0000","last_clean":"2024-03-24T00:36:41.063340+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:36:41.063340+0000","last_undegraded":"2024-03-24T00:36:41.063340+0000","last_fullsized":"2024-03-24T00:36:41.063340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078500+0000","last_change":"2024-03-24T00:35:42.436405+0000","last_active":"2024-03-24T00:36:41.078500+0000","last_peered":"2024-03-24T00:36:41.078500+0000","last_clean":"2024-03-24T00:36:41.078500+0000","last_became_active":"2024-03-24T00:35:42.436022+0000","last_became_peered":"2024-03-24T00:35:42.436022+0000","last_unstale":"2024-03-24T00:36:41.078500+0000","last_undegraded":"2024-03-24T00:36:41.078500+0000","last_fullsized":"2024-03-24T00:36:41.078500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:59.022152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063157+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:36:41.063157+0000","last_peered":"2024-03-24T00:36:41.063157+0000","last_clean":"2024-03-24T00:36:41.063157+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:36:41.063157+0000","last_undegraded":"2024-03-24T00:36:41.063157+0000","last_fullsized":"2024-03-24T00:36:41.063157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.046590+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:36:41.046590+0000","last_peered":"2024-03-24T00:36:41.046590+0000","last_clean":"2024-03-24T00:36:41.046590+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:36:41.046590+0000","last_undegraded":"2024-03-24T00:36:41.046590+0000","last_fullsized":"2024-03-24T00:36:41.046590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.077839+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:36:41.077839+0000","last_peered":"2024-03-24T00:36:41.077839+0000","last_clean":"2024-03-24T00:36:41.077839+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:36:41.077839+0000","last_undegraded":"2024-03-24T00:36:41.077839+0000","last_fullsized":"2024-03-24T00:36:41.077839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.084192+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:36:41.084192+0000","last_peered":"2024-03-24T00:36:41.084192+0000","last_clean":"2024-03-24T00:36:41.084192+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:36:41.084192+0000","last_undegraded":"2024-03-24T00:36:41.084192+0000","last_fullsized":"2024-03-24T00:36:41.084192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083963+0000","last_change":"2024-03-24T00:35:42.433339+0000","last_active":"2024-03-24T00:36:41.083963+0000","last_peered":"2024-03-24T00:36:41.083963+0000","last_clean":"2024-03-24T00:36:41.083963+0000","last_became_active":"2024-03-24T00:35:42.432819+0000","last_became_peered":"2024-03-24T00:35:42.432819+0000","last_unstale":"2024-03-24T00:36:41.083963+0000","last_undegraded":"2024-03-24T00:36:41.083963+0000","last_fullsized":"2024-03-24T00:36:41.083963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:09.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083784+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:36:41.083784+0000","last_peered":"2024-03-24T00:36:41.083784+0000","last_clean":"2024-03-24T00:36:41.083784+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:36:41.083784+0000","last_undegraded":"2024-03-24T00:36:41.083784+0000","last_fullsized":"2024-03-24T00:36:41.083784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082861+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:36:41.082861+0000","last_peered":"2024-03-24T00:36:41.082861+0000","last_clean":"2024-03-24T00:36:41.082861+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:36:41.082861+0000","last_undegraded":"2024-03-24T00:36:41.082861+0000","last_fullsized":"2024-03-24T00:36:41.082861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.079675+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:36:41.079675+0000","last_peered":"2024-03-24T00:36:41.079675+0000","last_clean":"2024-03-24T00:36:41.079675+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:36:41.079675+0000","last_undegraded":"2024-03-24T00:36:41.079675+0000","last_fullsized":"2024-03-24T00:36:41.079675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056950+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:36:41.056950+0000","last_peered":"2024-03-24T00:36:41.056950+0000","last_clean":"2024-03-24T00:36:41.056950+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:36:41.056950+0000","last_undegraded":"2024-03-24T00:36:41.056950+0000","last_fullsized":"2024-03-24T00:36:41.056950+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435499+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:42.435499+0000","last_peered":"2024-03-24T00:35:42.435499+0000","last_clean":"2024-03-24T00:35:42.435499+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:42.435499+0000","last_undegraded":"2024-03-24T00:35:42.435499+0000","last_fullsized":"2024-03-24T00:35:42.435499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063155+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:36:41.063155+0000","last_peered":"2024-03-24T00:36:41.063155+0000","last_clean":"2024-03-24T00:36:41.063155+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:36:41.063155+0000","last_undegraded":"2024-03-24T00:36:41.063155+0000","last_fullsized":"2024-03-24T00:36:41.063155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.433361+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:42.433361+0000","last_peered":"2024-03-24T00:35:42.433361+0000","last_clean":"2024-03-24T00:35:42.433361+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:42.433361+0000","last_undegraded":"2024-03-24T00:35:42.433361+0000","last_fullsized":"2024-03-24T00:35:42.433361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080915+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:36:41.080915+0000","last_peered":"2024-03-24T00:36:41.080915+0000","last_clean":"2024-03-24T00:36:41.080915+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:36:41.080915+0000","last_undegraded":"2024-03-24T00:36:41.080915+0000","last_fullsized":"2024-03-24T00:36:41.080915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435536+0000","last_change":"2024-03-24T00:35:42.435006+0000","last_active":"2024-03-24T00:35:42.435536+0000","last_peered":"2024-03-24T00:35:42.435536+0000","last_clean":"2024-03-24T00:35:42.435536+0000","last_became_active":"2024-03-24T00:35:42.429558+0000","last_became_peered":"2024-03-24T00:35:42.429558+0000","last_unstale":"2024-03-24T00:35:42.435536+0000","last_undegraded":"2024-03-24T00:35:42.435536+0000","last_fullsized":"2024-03-24T00:35:42.435536+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:09.212462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061003+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:36:41.061003+0000","last_peered":"2024-03-24T00:36:41.061003+0000","last_clean":"2024-03-24T00:36:41.061003+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:36:41.061003+0000","last_undegraded":"2024-03-24T00:36:41.061003+0000","last_fullsized":"2024-03-24T00:36:41.061003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431452+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:42.431452+0000","last_peered":"2024-03-24T00:35:42.431452+0000","last_clean":"2024-03-24T00:35:42.431452+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:42.431452+0000","last_undegraded":"2024-03-24T00:35:42.431452+0000","last_fullsized":"2024-03-24T00:35:42.431452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431007+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:42.431007+0000","last_peered":"2024-03-24T00:35:42.431007+0000","last_clean":"2024-03-24T00:35:42.431007+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:42.431007+0000","last_undegraded":"2024-03-24T00:35:42.431007+0000","last_fullsized":"2024-03-24T00:35:42.431007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074671+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:36:41.074671+0000","last_peered":"2024-03-24T00:36:41.074671+0000","last_clean":"2024-03-24T00:36:41.074671+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:36:41.074671+0000","last_undegraded":"2024-03-24T00:36:41.074671+0000","last_fullsized":"2024-03-24T00:36:41.074671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422152+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:42.422152+0000","last_peered":"2024-03-24T00:35:42.422152+0000","last_clean":"2024-03-24T00:35:42.422152+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:42.422152+0000","last_undegraded":"2024-03-24T00:35:42.422152+0000","last_fullsized":"2024-03-24T00:35:42.422152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061844+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:36:41.061844+0000","last_peered":"2024-03-24T00:36:41.061844+0000","last_clean":"2024-03-24T00:36:41.061844+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:36:41.061844+0000","last_undegraded":"2024-03-24T00:36:41.061844+0000","last_fullsized":"2024-03-24T00:36:41.061844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422989+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:42.422989+0000","last_peered":"2024-03-24T00:35:42.422989+0000","last_clean":"2024-03-24T00:35:42.422989+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:42.422989+0000","last_undegraded":"2024-03-24T00:35:42.422989+0000","last_fullsized":"2024-03-24T00:35:42.422989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071616+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:36:41.071616+0000","last_peered":"2024-03-24T00:36:41.071616+0000","last_clean":"2024-03-24T00:36:41.071616+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:36:41.071616+0000","last_undegraded":"2024-03-24T00:36:41.071616+0000","last_fullsized":"2024-03-24T00:36:41.071616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430389+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:42.430389+0000","last_peered":"2024-03-24T00:35:42.430389+0000","last_clean":"2024-03-24T00:35:42.430389+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:42.430389+0000","last_undegraded":"2024-03-24T00:35:42.430389+0000","last_fullsized":"2024-03-24T00:35:42.430389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073853+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:36:41.073853+0000","last_peered":"2024-03-24T00:36:41.073853+0000","last_clean":"2024-03-24T00:36:41.073853+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:36:41.073853+0000","last_undegraded":"2024-03-24T00:36:41.073853+0000","last_fullsized":"2024-03-24T00:36:41.073853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422441+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:42.422441+0000","last_peered":"2024-03-24T00:35:42.422441+0000","last_clean":"2024-03-24T00:35:42.422441+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:42.422441+0000","last_undegraded":"2024-03-24T00:35:42.422441+0000","last_fullsized":"2024-03-24T00:35:42.422441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078541+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:36:41.078541+0000","last_peered":"2024-03-24T00:36:41.078541+0000","last_clean":"2024-03-24T00:36:41.078541+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:36:41.078541+0000","last_undegraded":"2024-03-24T00:36:41.078541+0000","last_fullsized":"2024-03-24T00:36:41.078541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423580+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:42.423580+0000","last_peered":"2024-03-24T00:35:42.423580+0000","last_clean":"2024-03-24T00:35:42.423580+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:42.423580+0000","last_undegraded":"2024-03-24T00:35:42.423580+0000","last_fullsized":"2024-03-24T00:35:42.423580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083152+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:36:41.083152+0000","last_peered":"2024-03-24T00:36:41.083152+0000","last_clean":"2024-03-24T00:36:41.083152+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:36:41.083152+0000","last_undegraded":"2024-03-24T00:36:41.083152+0000","last_fullsized":"2024-03-24T00:36:41.083152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431398+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:42.431398+0000","last_peered":"2024-03-24T00:35:42.431398+0000","last_clean":"2024-03-24T00:35:42.431398+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:42.431398+0000","last_undegraded":"2024-03-24T00:35:42.431398+0000","last_fullsized":"2024-03-24T00:35:42.431398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057836+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:36:41.057836+0000","last_peered":"2024-03-24T00:36:41.057836+0000","last_clean":"2024-03-24T00:36:41.057836+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:36:41.057836+0000","last_undegraded":"2024-03-24T00:36:41.057836+0000","last_fullsized":"2024-03-24T00:36:41.057836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":341,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:54.321438+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:37:54.321438+0000","last_peered":"2024-03-24T00:37:54.321438+0000","last_clean":"2024-03-24T00:37:54.321438+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:37:54.321438+0000","last_undegraded":"2024-03-24T00:37:54.321438+0000","last_fullsized":"2024-03-24T00:37:54.321438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67824,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":289,"num_read_kb":211,"num_write":32,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062432+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:36:41.062432+0000","last_peered":"2024-03-24T00:36:41.062432+0000","last_clean":"2024-03-24T00:36:41.062432+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:36:41.062432+0000","last_undegraded":"2024-03-24T00:36:41.062432+0000","last_fullsized":"2024-03-24T00:36:41.062432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424521+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:42.424521+0000","last_peered":"2024-03-24T00:35:42.424521+0000","last_clean":"2024-03-24T00:35:42.424521+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:42.424521+0000","last_undegraded":"2024-03-24T00:35:42.424521+0000","last_fullsized":"2024-03-24T00:35:42.424521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067885+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:36:41.067885+0000","last_peered":"2024-03-24T00:36:41.067885+0000","last_clean":"2024-03-24T00:36:41.067885+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:36:41.067885+0000","last_undegraded":"2024-03-24T00:36:41.067885+0000","last_fullsized":"2024-03-24T00:36:41.067885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430861+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:42.430861+0000","last_peered":"2024-03-24T00:35:42.430861+0000","last_clean":"2024-03-24T00:35:42.430861+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:42.430861+0000","last_undegraded":"2024-03-24T00:35:42.430861+0000","last_fullsized":"2024-03-24T00:35:42.430861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078591+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:36:41.078591+0000","last_peered":"2024-03-24T00:36:41.078591+0000","last_clean":"2024-03-24T00:36:41.078591+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:36:41.078591+0000","last_undegraded":"2024-03-24T00:36:41.078591+0000","last_fullsized":"2024-03-24T00:36:41.078591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425560+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:42.425560+0000","last_peered":"2024-03-24T00:35:42.425560+0000","last_clean":"2024-03-24T00:35:42.425560+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:42.425560+0000","last_undegraded":"2024-03-24T00:35:42.425560+0000","last_fullsized":"2024-03-24T00:35:42.425560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067539+0000","last_change":"2024-03-24T00:35:42.436171+0000","last_active":"2024-03-24T00:36:41.067539+0000","last_peered":"2024-03-24T00:36:41.067539+0000","last_clean":"2024-03-24T00:36:41.067539+0000","last_became_active":"2024-03-24T00:35:42.435757+0000","last_became_peered":"2024-03-24T00:35:42.435757+0000","last_unstale":"2024-03-24T00:36:41.067539+0000","last_undegraded":"2024-03-24T00:36:41.067539+0000","last_fullsized":"2024-03-24T00:36:41.067539+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:08.194563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430666+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:42.430666+0000","last_peered":"2024-03-24T00:35:42.430666+0000","last_clean":"2024-03-24T00:35:42.430666+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:42.430666+0000","last_undegraded":"2024-03-24T00:35:42.430666+0000","last_fullsized":"2024-03-24T00:35:42.430666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059832+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:36:41.059832+0000","last_peered":"2024-03-24T00:36:41.059832+0000","last_clean":"2024-03-24T00:36:41.059832+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:36:41.059832+0000","last_undegraded":"2024-03-24T00:36:41.059832+0000","last_fullsized":"2024-03-24T00:36:41.059832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.438512+0000","last_change":"2024-03-24T00:35:42.438099+0000","last_active":"2024-03-24T00:35:42.438512+0000","last_peered":"2024-03-24T00:35:42.438512+0000","last_clean":"2024-03-24T00:35:42.438512+0000","last_became_active":"2024-03-24T00:35:42.432549+0000","last_became_peered":"2024-03-24T00:35:42.432549+0000","last_unstale":"2024-03-24T00:35:42.438512+0000","last_undegraded":"2024-03-24T00:35:42.438512+0000","last_fullsized":"2024-03-24T00:35:42.438512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:33.579265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056704+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:36:41.056704+0000","last_peered":"2024-03-24T00:36:41.056704+0000","last_clean":"2024-03-24T00:36:41.056704+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:36:41.056704+0000","last_undegraded":"2024-03-24T00:36:41.056704+0000","last_fullsized":"2024-03-24T00:36:41.056704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.434738+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:42.434738+0000","last_peered":"2024-03-24T00:35:42.434738+0000","last_clean":"2024-03-24T00:35:42.434738+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:42.434738+0000","last_undegraded":"2024-03-24T00:35:42.434738+0000","last_fullsized":"2024-03-24T00:35:42.434738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.049339+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:36:41.049339+0000","last_peered":"2024-03-24T00:36:41.049339+0000","last_clean":"2024-03-24T00:36:41.049339+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:36:41.049339+0000","last_undegraded":"2024-03-24T00:36:41.049339+0000","last_fullsized":"2024-03-24T00:36:41.049339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423562+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:42.423562+0000","last_peered":"2024-03-24T00:35:42.423562+0000","last_clean":"2024-03-24T00:35:42.423562+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:42.423562+0000","last_undegraded":"2024-03-24T00:35:42.423562+0000","last_fullsized":"2024-03-24T00:35:42.423562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082291+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:36:41.082291+0000","last_peered":"2024-03-24T00:36:41.082291+0000","last_clean":"2024-03-24T00:36:41.082291+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:36:41.082291+0000","last_undegraded":"2024-03-24T00:36:41.082291+0000","last_fullsized":"2024-03-24T00:36:41.082291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.051541+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:36:41.051541+0000","last_peered":"2024-03-24T00:36:41.051541+0000","last_clean":"2024-03-24T00:36:41.051541+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:36:41.051541+0000","last_undegraded":"2024-03-24T00:36:41.051541+0000","last_fullsized":"2024-03-24T00:36:41.051541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422997+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:42.422997+0000","last_peered":"2024-03-24T00:35:42.422997+0000","last_clean":"2024-03-24T00:35:42.422997+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:42.422997+0000","last_undegraded":"2024-03-24T00:35:42.422997+0000","last_fullsized":"2024-03-24T00:35:42.422997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064055+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:36:41.064055+0000","last_peered":"2024-03-24T00:36:41.064055+0000","last_clean":"2024-03-24T00:36:41.064055+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:36:41.064055+0000","last_undegraded":"2024-03-24T00:36:41.064055+0000","last_fullsized":"2024-03-24T00:36:41.064055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425657+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:42.425657+0000","last_peered":"2024-03-24T00:35:42.425657+0000","last_clean":"2024-03-24T00:35:42.425657+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:42.425657+0000","last_undegraded":"2024-03-24T00:35:42.425657+0000","last_fullsized":"2024-03-24T00:35:42.425657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423432+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:42.423432+0000","last_peered":"2024-03-24T00:35:42.423432+0000","last_clean":"2024-03-24T00:35:42.423432+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:42.423432+0000","last_undegraded":"2024-03-24T00:35:42.423432+0000","last_fullsized":"2024-03-24T00:35:42.423432+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060829+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:36:41.060829+0000","last_peered":"2024-03-24T00:36:41.060829+0000","last_clean":"2024-03-24T00:36:41.060829+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:36:41.060829+0000","last_undegraded":"2024-03-24T00:36:41.060829+0000","last_fullsized":"2024-03-24T00:36:41.060829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423329+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:42.423329+0000","last_peered":"2024-03-24T00:35:42.423329+0000","last_clean":"2024-03-24T00:35:42.423329+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:42.423329+0000","last_undegraded":"2024-03-24T00:35:42.423329+0000","last_fullsized":"2024-03-24T00:35:42.423329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431514+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:42.431514+0000","last_peered":"2024-03-24T00:35:42.431514+0000","last_clean":"2024-03-24T00:35:42.431514+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:42.431514+0000","last_undegraded":"2024-03-24T00:35:42.431514+0000","last_fullsized":"2024-03-24T00:35:42.431514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074782+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:36:41.074782+0000","last_peered":"2024-03-24T00:36:41.074782+0000","last_clean":"2024-03-24T00:36:41.074782+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:36:41.074782+0000","last_undegraded":"2024-03-24T00:36:41.074782+0000","last_fullsized":"2024-03-24T00:36:41.074782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:42.954936+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:37:42.954936+0000","last_peered":"2024-03-24T00:37:42.954936+0000","last_clean":"2024-03-24T00:37:42.954936+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:37:42.954936+0000","last_undegraded":"2024-03-24T00:37:42.954936+0000","last_fullsized":"2024-03-24T00:37:42.954936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066366+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:36:41.066366+0000","last_peered":"2024-03-24T00:36:41.066366+0000","last_clean":"2024-03-24T00:36:41.066366+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:36:41.066366+0000","last_undegraded":"2024-03-24T00:36:41.066366+0000","last_fullsized":"2024-03-24T00:36:41.066366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425950+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:42.425950+0000","last_peered":"2024-03-24T00:35:42.425950+0000","last_clean":"2024-03-24T00:35:42.425950+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:42.425950+0000","last_undegraded":"2024-03-24T00:35:42.425950+0000","last_fullsized":"2024-03-24T00:35:42.425950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075408+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:36:41.075408+0000","last_peered":"2024-03-24T00:36:41.075408+0000","last_clean":"2024-03-24T00:36:41.075408+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:36:41.075408+0000","last_undegraded":"2024-03-24T00:36:41.075408+0000","last_fullsized":"2024-03-24T00:36:41.075408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.862689+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:42.862689+0000","last_peered":"2024-03-24T00:35:42.862689+0000","last_clean":"2024-03-24T00:35:42.862689+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:42.862689+0000","last_undegraded":"2024-03-24T00:35:42.862689+0000","last_fullsized":"2024-03-24T00:35:42.862689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062126+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:36:41.062126+0000","last_peered":"2024-03-24T00:36:41.062126+0000","last_clean":"2024-03-24T00:36:41.062126+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:36:41.062126+0000","last_undegraded":"2024-03-24T00:36:41.062126+0000","last_fullsized":"2024-03-24T00:36:41.062126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430960+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:42.430960+0000","last_peered":"2024-03-24T00:35:42.430960+0000","last_clean":"2024-03-24T00:35:42.430960+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:42.430960+0000","last_undegraded":"2024-03-24T00:35:42.430960+0000","last_fullsized":"2024-03-24T00:35:42.430960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082706+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:36:41.082706+0000","last_peered":"2024-03-24T00:36:41.082706+0000","last_clean":"2024-03-24T00:36:41.082706+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:36:41.082706+0000","last_undegraded":"2024-03-24T00:36:41.082706+0000","last_fullsized":"2024-03-24T00:36:41.082706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424573+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:42.424573+0000","last_peered":"2024-03-24T00:35:42.424573+0000","last_clean":"2024-03-24T00:35:42.424573+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:42.424573+0000","last_undegraded":"2024-03-24T00:35:42.424573+0000","last_fullsized":"2024-03-24T00:35:42.424573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080662+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:36:41.080662+0000","last_peered":"2024-03-24T00:36:41.080662+0000","last_clean":"2024-03-24T00:36:41.080662+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:36:41.080662+0000","last_undegraded":"2024-03-24T00:36:41.080662+0000","last_fullsized":"2024-03-24T00:36:41.080662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430784+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:42.430784+0000","last_peered":"2024-03-24T00:35:42.430784+0000","last_clean":"2024-03-24T00:35:42.430784+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:42.430784+0000","last_undegraded":"2024-03-24T00:35:42.430784+0000","last_fullsized":"2024-03-24T00:35:42.430784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071515+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:36:41.071515+0000","last_peered":"2024-03-24T00:36:41.071515+0000","last_clean":"2024-03-24T00:36:41.071515+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:36:41.071515+0000","last_undegraded":"2024-03-24T00:36:41.071515+0000","last_fullsized":"2024-03-24T00:36:41.071515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.432672+0000","last_change":"2024-03-24T00:35:42.432365+0000","last_active":"2024-03-24T00:35:42.432672+0000","last_peered":"2024-03-24T00:35:42.432672+0000","last_clean":"2024-03-24T00:35:42.432672+0000","last_became_active":"2024-03-24T00:35:42.432113+0000","last_became_peered":"2024-03-24T00:35:42.432113+0000","last_unstale":"2024-03-24T00:35:42.432672+0000","last_undegraded":"2024-03-24T00:35:42.432672+0000","last_fullsized":"2024-03-24T00:35:42.432672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:32.467534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065534+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:36:41.065534+0000","last_peered":"2024-03-24T00:36:41.065534+0000","last_clean":"2024-03-24T00:36:41.065534+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:36:41.065534+0000","last_undegraded":"2024-03-24T00:36:41.065534+0000","last_fullsized":"2024-03-24T00:36:41.065534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431250+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:42.431250+0000","last_peered":"2024-03-24T00:35:42.431250+0000","last_clean":"2024-03-24T00:35:42.431250+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:42.431250+0000","last_undegraded":"2024-03-24T00:35:42.431250+0000","last_fullsized":"2024-03-24T00:35:42.431250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066395+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:36:41.066395+0000","last_peered":"2024-03-24T00:36:41.066395+0000","last_clean":"2024-03-24T00:36:41.066395+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:36:41.066395+0000","last_undegraded":"2024-03-24T00:36:41.066395+0000","last_fullsized":"2024-03-24T00:36:41.066395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.867566+0000","last_change":"2024-03-24T00:35:42.866804+0000","last_active":"2024-03-24T00:35:42.867566+0000","last_peered":"2024-03-24T00:35:42.867566+0000","last_clean":"2024-03-24T00:35:42.867566+0000","last_became_active":"2024-03-24T00:35:42.863933+0000","last_became_peered":"2024-03-24T00:35:42.863933+0000","last_unstale":"2024-03-24T00:35:42.867566+0000","last_undegraded":"2024-03-24T00:35:42.867566+0000","last_fullsized":"2024-03-24T00:35:42.867566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:55.475527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062309+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:36:41.062309+0000","last_peered":"2024-03-24T00:36:41.062309+0000","last_clean":"2024-03-24T00:36:41.062309+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:36:41.062309+0000","last_undegraded":"2024-03-24T00:36:41.062309+0000","last_fullsized":"2024-03-24T00:36:41.062309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061017+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:36:41.061017+0000","last_peered":"2024-03-24T00:36:41.061017+0000","last_clean":"2024-03-24T00:36:41.061017+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:36:41.061017+0000","last_undegraded":"2024-03-24T00:36:41.061017+0000","last_fullsized":"2024-03-24T00:36:41.061017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.053820+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:36:41.053820+0000","last_peered":"2024-03-24T00:36:41.053820+0000","last_clean":"2024-03-24T00:36:41.053820+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:36:41.053820+0000","last_undegraded":"2024-03-24T00:36:41.053820+0000","last_fullsized":"2024-03-24T00:36:41.053820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074338+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:36:41.074338+0000","last_peered":"2024-03-24T00:36:41.074338+0000","last_clean":"2024-03-24T00:36:41.074338+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:36:41.074338+0000","last_undegraded":"2024-03-24T00:36:41.074338+0000","last_fullsized":"2024-03-24T00:36:41.074338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080606+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:36:41.080606+0000","last_peered":"2024-03-24T00:36:41.080606+0000","last_clean":"2024-03-24T00:36:41.080606+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:36:41.080606+0000","last_undegraded":"2024-03-24T00:36:41.080606+0000","last_fullsized":"2024-03-24T00:36:41.080606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.076656+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:36:41.076656+0000","last_peered":"2024-03-24T00:36:41.076656+0000","last_clean":"2024-03-24T00:36:41.076656+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:36:41.076656+0000","last_undegraded":"2024-03-24T00:36:41.076656+0000","last_fullsized":"2024-03-24T00:36:41.076656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056546+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:36:41.056546+0000","last_peered":"2024-03-24T00:36:41.056546+0000","last_clean":"2024-03-24T00:36:41.056546+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:36:41.056546+0000","last_undegraded":"2024-03-24T00:36:41.056546+0000","last_fullsized":"2024-03-24T00:36:41.056546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061532+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:36:41.061532+0000","last_peered":"2024-03-24T00:36:41.061532+0000","last_clean":"2024-03-24T00:36:41.061532+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:36:41.061532+0000","last_undegraded":"2024-03-24T00:36:41.061532+0000","last_fullsized":"2024-03-24T00:36:41.061532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.069575+0000","last_change":"2024-03-24T00:35:42.437175+0000","last_active":"2024-03-24T00:36:41.069575+0000","last_peered":"2024-03-24T00:36:41.069575+0000","last_clean":"2024-03-24T00:36:41.069575+0000","last_became_active":"2024-03-24T00:35:42.436625+0000","last_became_peered":"2024-03-24T00:35:42.436625+0000","last_unstale":"2024-03-24T00:36:41.069575+0000","last_undegraded":"2024-03-24T00:36:41.069575+0000","last_fullsized":"2024-03-24T00:36:41.069575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:29:26.922046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066129+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:36:41.066129+0000","last_peered":"2024-03-24T00:36:41.066129+0000","last_clean":"2024-03-24T00:36:41.066129+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:36:41.066129+0000","last_undegraded":"2024-03-24T00:36:41.066129+0000","last_fullsized":"2024-03-24T00:36:41.066129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064052+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:36:41.064052+0000","last_peered":"2024-03-24T00:36:41.064052+0000","last_clean":"2024-03-24T00:36:41.064052+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:36:41.064052+0000","last_undegraded":"2024-03-24T00:36:41.064052+0000","last_fullsized":"2024-03-24T00:36:41.064052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064779+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:36:41.064779+0000","last_peered":"2024-03-24T00:36:41.064779+0000","last_clean":"2024-03-24T00:36:41.064779+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:36:41.064779+0000","last_undegraded":"2024-03-24T00:36:41.064779+0000","last_fullsized":"2024-03-24T00:36:41.064779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.081897+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:36:41.081897+0000","last_peered":"2024-03-24T00:36:41.081897+0000","last_clean":"2024-03-24T00:36:41.081897+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:36:41.081897+0000","last_undegraded":"2024-03-24T00:36:41.081897+0000","last_fullsized":"2024-03-24T00:36:41.081897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057088+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:36:41.057088+0000","last_peered":"2024-03-24T00:36:41.057088+0000","last_clean":"2024-03-24T00:36:41.057088+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:36:41.057088+0000","last_undegraded":"2024-03-24T00:36:41.057088+0000","last_fullsized":"2024-03-24T00:36:41.057088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68622,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":294,"num_read_kb":216,"num_write":89,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":200187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579952,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1646592,"data_stored":1124419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1056983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":528367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":528095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":466254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724619,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1056583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:55.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 -- 172.21.15.123:0/1613304095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 msgr2=0x7f21f4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1613304095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f4104f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 -- 172.21.15.123:0/1613304095 shutdown_connections 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1613304095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 0x7f21f410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1613304095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f4104f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1613304095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 -- 172.21.15.123:0/1613304095 >> 172.21.15.123:0/1613304095 conn(0x7f21f40fee90 msgr2=0x7f21f41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 -- 172.21.15.123:0/1613304095 shutdown_connections 2024-03-24T00:37:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.244+0000 7f21f9e00640 1 -- 172.21.15.123:0/1613304095 wait complete. 2024-03-24T00:37:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.245+0000 7f21f9e00640 1 Processor -- start 2024-03-24T00:37:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.245+0000 7f21f9e00640 1 -- start start 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.245+0000 7f21f9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 0x7f21f41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410f560 con 0x7f21f41038e0 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410f3e0 con 0x7f21f4104ad0 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f9e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410f6e0 con 0x7f21f4105490 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45460/0 (socket says 172.21.15.123:45460) 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f2a00640 1 -- 172.21.15.123:0/1981197513 learned_addr learned my addr 172.21.15.123:0/1981197513 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21f3400640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 0x7f21f41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 -- 172.21.15.123:0/1981197513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 msgr2=0x7f21f41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 0x7f21f41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 -- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 msgr2=0x7f21f4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.246+0000 7f21ea000640 1 -- 172.21.15.123:0/1981197513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f41a17a0 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21ea000640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f419a0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009230 tx=0x7f21dc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc041070 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21dc004450 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f41a1a30 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc003df0 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.247+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f41a1f70 con 0x7f21f4104ad0 2024-03-24T00:37:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.248+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f410aba0 con 0x7f21f4104ad0 2024-03-24T00:37:56.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.250+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f21dc044070 con 0x7f21f4104ad0 2024-03-24T00:37:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.251+0000 7f21f8a00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.251+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f21dc0f0510 con 0x7f21f4104ad0 2024-03-24T00:37:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.251+0000 7f21f2a00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.252+0000 7f21f2a00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 0x7f21d80a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f21e4011440 tx=0x7f21e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.256+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21dc0bd560 con 0x7f21f4104ad0 2024-03-24T00:37:56.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.456+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21f4061ae0 con 0x7f21d80a2880 2024-03-24T00:37:56.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.468+0000 7f21f8a00640 1 -- 172.21.15.123:0/1981197513 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258757 (secure 0 0 0) 0x7f21f4061ae0 con 0x7f21d80a2880 2024-03-24T00:37:56.472 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:56.473 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:37:56.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.472+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 msgr2=0x7f21d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.472+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 0x7f21d80a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f21e4011440 tx=0x7f21e4009500 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.472+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 msgr2=0x7f21f419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.472+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f419a0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009230 tx=0x7f21dc02c650 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 shutdown_connections 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21f4105490 0x7f21f41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f4104ad0 0x7f21f419a0c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 --2- 172.21.15.123:0/1981197513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f41038e0 0x7f21f4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 >> 172.21.15.123:0/1981197513 conn(0x7f21f40fee90 msgr2=0x7f21f4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 shutdown_connections 2024-03-24T00:37:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.473+0000 7f21f9e00640 1 -- 172.21.15.123:0/1981197513 wait complete. 2024-03-24T00:37:56.493 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-03-24T00:37:55.125464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659224,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":420,"num_read_kb":325,"num_write":272,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354108,"kb_used_data":13276,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587716,"statfs":{"total":1151940427776,"available":1151577821184,"internally_reserved":0,"allocated":13594624,"data_stored":7557612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5679,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004698"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082792+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:36:41.082792+0000","last_peered":"2024-03-24T00:36:41.082792+0000","last_clean":"2024-03-24T00:36:41.082792+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:36:41.082792+0000","last_undegraded":"2024-03-24T00:36:41.082792+0000","last_fullsized":"2024-03-24T00:36:41.082792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083200+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:36:41.083200+0000","last_peered":"2024-03-24T00:36:41.083200+0000","last_clean":"2024-03-24T00:36:41.083200+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:36:41.083200+0000","last_undegraded":"2024-03-24T00:36:41.083200+0000","last_fullsized":"2024-03-24T00:36:41.083200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082591+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:36:41.082591+0000","last_peered":"2024-03-24T00:36:41.082591+0000","last_clean":"2024-03-24T00:36:41.082591+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:36:41.082591+0000","last_undegraded":"2024-03-24T00:36:41.082591+0000","last_fullsized":"2024-03-24T00:36:41.082591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059003+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:36:41.059003+0000","last_peered":"2024-03-24T00:36:41.059003+0000","last_clean":"2024-03-24T00:36:41.059003+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:36:41.059003+0000","last_undegraded":"2024-03-24T00:36:41.059003+0000","last_fullsized":"2024-03-24T00:36:41.059003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.068426+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:36:41.068426+0000","last_peered":"2024-03-24T00:36:41.068426+0000","last_clean":"2024-03-24T00:36:41.068426+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:36:41.068426+0000","last_undegraded":"2024-03-24T00:36:41.068426+0000","last_fullsized":"2024-03-24T00:36:41.068426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073848+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:36:41.073848+0000","last_peered":"2024-03-24T00:36:41.073848+0000","last_clean":"2024-03-24T00:36:41.073848+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:36:41.073848+0000","last_undegraded":"2024-03-24T00:36:41.073848+0000","last_fullsized":"2024-03-24T00:36:41.073848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074753+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:36:41.074753+0000","last_peered":"2024-03-24T00:36:41.074753+0000","last_clean":"2024-03-24T00:36:41.074753+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:36:41.074753+0000","last_undegraded":"2024-03-24T00:36:41.074753+0000","last_fullsized":"2024-03-24T00:36:41.074753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065233+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:36:41.065233+0000","last_peered":"2024-03-24T00:36:41.065233+0000","last_clean":"2024-03-24T00:36:41.065233+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:36:41.065233+0000","last_undegraded":"2024-03-24T00:36:41.065233+0000","last_fullsized":"2024-03-24T00:36:41.065233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063340+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:36:41.063340+0000","last_peered":"2024-03-24T00:36:41.063340+0000","last_clean":"2024-03-24T00:36:41.063340+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:36:41.063340+0000","last_undegraded":"2024-03-24T00:36:41.063340+0000","last_fullsized":"2024-03-24T00:36:41.063340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078500+0000","last_change":"2024-03-24T00:35:42.436405+0000","last_active":"2024-03-24T00:36:41.078500+0000","last_peered":"2024-03-24T00:36:41.078500+0000","last_clean":"2024-03-24T00:36:41.078500+0000","last_became_active":"2024-03-24T00:35:42.436022+0000","last_became_peered":"2024-03-24T00:35:42.436022+0000","last_unstale":"2024-03-24T00:36:41.078500+0000","last_undegraded":"2024-03-24T00:36:41.078500+0000","last_fullsized":"2024-03-24T00:36:41.078500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:59.022152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063157+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:36:41.063157+0000","last_peered":"2024-03-24T00:36:41.063157+0000","last_clean":"2024-03-24T00:36:41.063157+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:36:41.063157+0000","last_undegraded":"2024-03-24T00:36:41.063157+0000","last_fullsized":"2024-03-24T00:36:41.063157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.046590+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:36:41.046590+0000","last_peered":"2024-03-24T00:36:41.046590+0000","last_clean":"2024-03-24T00:36:41.046590+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:36:41.046590+0000","last_undegraded":"2024-03-24T00:36:41.046590+0000","last_fullsized":"2024-03-24T00:36:41.046590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.077839+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:36:41.077839+0000","last_peered":"2024-03-24T00:36:41.077839+0000","last_clean":"2024-03-24T00:36:41.077839+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:36:41.077839+0000","last_undegraded":"2024-03-24T00:36:41.077839+0000","last_fullsized":"2024-03-24T00:36:41.077839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.084192+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:36:41.084192+0000","last_peered":"2024-03-24T00:36:41.084192+0000","last_clean":"2024-03-24T00:36:41.084192+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:36:41.084192+0000","last_undegraded":"2024-03-24T00:36:41.084192+0000","last_fullsized":"2024-03-24T00:36:41.084192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083963+0000","last_change":"2024-03-24T00:35:42.433339+0000","last_active":"2024-03-24T00:36:41.083963+0000","last_peered":"2024-03-24T00:36:41.083963+0000","last_clean":"2024-03-24T00:36:41.083963+0000","last_became_active":"2024-03-24T00:35:42.432819+0000","last_became_peered":"2024-03-24T00:35:42.432819+0000","last_unstale":"2024-03-24T00:36:41.083963+0000","last_undegraded":"2024-03-24T00:36:41.083963+0000","last_fullsized":"2024-03-24T00:36:41.083963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:09.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083784+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:36:41.083784+0000","last_peered":"2024-03-24T00:36:41.083784+0000","last_clean":"2024-03-24T00:36:41.083784+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:36:41.083784+0000","last_undegraded":"2024-03-24T00:36:41.083784+0000","last_fullsized":"2024-03-24T00:36:41.083784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082861+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:36:41.082861+0000","last_peered":"2024-03-24T00:36:41.082861+0000","last_clean":"2024-03-24T00:36:41.082861+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:36:41.082861+0000","last_undegraded":"2024-03-24T00:36:41.082861+0000","last_fullsized":"2024-03-24T00:36:41.082861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.079675+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:36:41.079675+0000","last_peered":"2024-03-24T00:36:41.079675+0000","last_clean":"2024-03-24T00:36:41.079675+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:36:41.079675+0000","last_undegraded":"2024-03-24T00:36:41.079675+0000","last_fullsized":"2024-03-24T00:36:41.079675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056950+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:36:41.056950+0000","last_peered":"2024-03-24T00:36:41.056950+0000","last_clean":"2024-03-24T00:36:41.056950+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:36:41.056950+0000","last_undegraded":"2024-03-24T00:36:41.056950+0000","last_fullsized":"2024-03-24T00:36:41.056950+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435499+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:42.435499+0000","last_peered":"2024-03-24T00:35:42.435499+0000","last_clean":"2024-03-24T00:35:42.435499+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:42.435499+0000","last_undegraded":"2024-03-24T00:35:42.435499+0000","last_fullsized":"2024-03-24T00:35:42.435499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063155+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:36:41.063155+0000","last_peered":"2024-03-24T00:36:41.063155+0000","last_clean":"2024-03-24T00:36:41.063155+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:36:41.063155+0000","last_undegraded":"2024-03-24T00:36:41.063155+0000","last_fullsized":"2024-03-24T00:36:41.063155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.433361+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:42.433361+0000","last_peered":"2024-03-24T00:35:42.433361+0000","last_clean":"2024-03-24T00:35:42.433361+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:42.433361+0000","last_undegraded":"2024-03-24T00:35:42.433361+0000","last_fullsized":"2024-03-24T00:35:42.433361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080915+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:36:41.080915+0000","last_peered":"2024-03-24T00:36:41.080915+0000","last_clean":"2024-03-24T00:36:41.080915+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:36:41.080915+0000","last_undegraded":"2024-03-24T00:36:41.080915+0000","last_fullsized":"2024-03-24T00:36:41.080915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435536+0000","last_change":"2024-03-24T00:35:42.435006+0000","last_active":"2024-03-24T00:35:42.435536+0000","last_peered":"2024-03-24T00:35:42.435536+0000","last_clean":"2024-03-24T00:35:42.435536+0000","last_became_active":"2024-03-24T00:35:42.429558+0000","last_became_peered":"2024-03-24T00:35:42.429558+0000","last_unstale":"2024-03-24T00:35:42.435536+0000","last_undegraded":"2024-03-24T00:35:42.435536+0000","last_fullsized":"2024-03-24T00:35:42.435536+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:09.212462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061003+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:36:41.061003+0000","last_peered":"2024-03-24T00:36:41.061003+0000","last_clean":"2024-03-24T00:36:41.061003+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:36:41.061003+0000","last_undegraded":"2024-03-24T00:36:41.061003+0000","last_fullsized":"2024-03-24T00:36:41.061003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431452+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:42.431452+0000","last_peered":"2024-03-24T00:35:42.431452+0000","last_clean":"2024-03-24T00:35:42.431452+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:42.431452+0000","last_undegraded":"2024-03-24T00:35:42.431452+0000","last_fullsized":"2024-03-24T00:35:42.431452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431007+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:42.431007+0000","last_peered":"2024-03-24T00:35:42.431007+0000","last_clean":"2024-03-24T00:35:42.431007+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:42.431007+0000","last_undegraded":"2024-03-24T00:35:42.431007+0000","last_fullsized":"2024-03-24T00:35:42.431007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074671+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:36:41.074671+0000","last_peered":"2024-03-24T00:36:41.074671+0000","last_clean":"2024-03-24T00:36:41.074671+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:36:41.074671+0000","last_undegraded":"2024-03-24T00:36:41.074671+0000","last_fullsized":"2024-03-24T00:36:41.074671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422152+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:42.422152+0000","last_peered":"2024-03-24T00:35:42.422152+0000","last_clean":"2024-03-24T00:35:42.422152+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:42.422152+0000","last_undegraded":"2024-03-24T00:35:42.422152+0000","last_fullsized":"2024-03-24T00:35:42.422152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061844+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:36:41.061844+0000","last_peered":"2024-03-24T00:36:41.061844+0000","last_clean":"2024-03-24T00:36:41.061844+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:36:41.061844+0000","last_undegraded":"2024-03-24T00:36:41.061844+0000","last_fullsized":"2024-03-24T00:36:41.061844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422989+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:42.422989+0000","last_peered":"2024-03-24T00:35:42.422989+0000","last_clean":"2024-03-24T00:35:42.422989+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:42.422989+0000","last_undegraded":"2024-03-24T00:35:42.422989+0000","last_fullsized":"2024-03-24T00:35:42.422989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071616+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:36:41.071616+0000","last_peered":"2024-03-24T00:36:41.071616+0000","last_clean":"2024-03-24T00:36:41.071616+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:36:41.071616+0000","last_undegraded":"2024-03-24T00:36:41.071616+0000","last_fullsized":"2024-03-24T00:36:41.071616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430389+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:42.430389+0000","last_peered":"2024-03-24T00:35:42.430389+0000","last_clean":"2024-03-24T00:35:42.430389+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:42.430389+0000","last_undegraded":"2024-03-24T00:35:42.430389+0000","last_fullsized":"2024-03-24T00:35:42.430389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073853+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:36:41.073853+0000","last_peered":"2024-03-24T00:36:41.073853+0000","last_clean":"2024-03-24T00:36:41.073853+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:36:41.073853+0000","last_undegraded":"2024-03-24T00:36:41.073853+0000","last_fullsized":"2024-03-24T00:36:41.073853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422441+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:42.422441+0000","last_peered":"2024-03-24T00:35:42.422441+0000","last_clean":"2024-03-24T00:35:42.422441+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:42.422441+0000","last_undegraded":"2024-03-24T00:35:42.422441+0000","last_fullsized":"2024-03-24T00:35:42.422441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078541+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:36:41.078541+0000","last_peered":"2024-03-24T00:36:41.078541+0000","last_clean":"2024-03-24T00:36:41.078541+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:36:41.078541+0000","last_undegraded":"2024-03-24T00:36:41.078541+0000","last_fullsized":"2024-03-24T00:36:41.078541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423580+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:42.423580+0000","last_peered":"2024-03-24T00:35:42.423580+0000","last_clean":"2024-03-24T00:35:42.423580+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:42.423580+0000","last_undegraded":"2024-03-24T00:35:42.423580+0000","last_fullsized":"2024-03-24T00:35:42.423580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083152+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:36:41.083152+0000","last_peered":"2024-03-24T00:36:41.083152+0000","last_clean":"2024-03-24T00:36:41.083152+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:36:41.083152+0000","last_undegraded":"2024-03-24T00:36:41.083152+0000","last_fullsized":"2024-03-24T00:36:41.083152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431398+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:42.431398+0000","last_peered":"2024-03-24T00:35:42.431398+0000","last_clean":"2024-03-24T00:35:42.431398+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:42.431398+0000","last_undegraded":"2024-03-24T00:35:42.431398+0000","last_fullsized":"2024-03-24T00:35:42.431398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057836+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:36:41.057836+0000","last_peered":"2024-03-24T00:36:41.057836+0000","last_clean":"2024-03-24T00:36:41.057836+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:36:41.057836+0000","last_undegraded":"2024-03-24T00:36:41.057836+0000","last_fullsized":"2024-03-24T00:36:41.057836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":341,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:54.321438+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:37:54.321438+0000","last_peered":"2024-03-24T00:37:54.321438+0000","last_clean":"2024-03-24T00:37:54.321438+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:37:54.321438+0000","last_undegraded":"2024-03-24T00:37:54.321438+0000","last_fullsized":"2024-03-24T00:37:54.321438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67824,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":289,"num_read_kb":211,"num_write":32,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062432+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:36:41.062432+0000","last_peered":"2024-03-24T00:36:41.062432+0000","last_clean":"2024-03-24T00:36:41.062432+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:36:41.062432+0000","last_undegraded":"2024-03-24T00:36:41.062432+0000","last_fullsized":"2024-03-24T00:36:41.062432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424521+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:42.424521+0000","last_peered":"2024-03-24T00:35:42.424521+0000","last_clean":"2024-03-24T00:35:42.424521+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:42.424521+0000","last_undegraded":"2024-03-24T00:35:42.424521+0000","last_fullsized":"2024-03-24T00:35:42.424521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067885+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:36:41.067885+0000","last_peered":"2024-03-24T00:36:41.067885+0000","last_clean":"2024-03-24T00:36:41.067885+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:36:41.067885+0000","last_undegraded":"2024-03-24T00:36:41.067885+0000","last_fullsized":"2024-03-24T00:36:41.067885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430861+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:42.430861+0000","last_peered":"2024-03-24T00:35:42.430861+0000","last_clean":"2024-03-24T00:35:42.430861+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:42.430861+0000","last_undegraded":"2024-03-24T00:35:42.430861+0000","last_fullsized":"2024-03-24T00:35:42.430861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078591+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:36:41.078591+0000","last_peered":"2024-03-24T00:36:41.078591+0000","last_clean":"2024-03-24T00:36:41.078591+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:36:41.078591+0000","last_undegraded":"2024-03-24T00:36:41.078591+0000","last_fullsized":"2024-03-24T00:36:41.078591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425560+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:42.425560+0000","last_peered":"2024-03-24T00:35:42.425560+0000","last_clean":"2024-03-24T00:35:42.425560+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:42.425560+0000","last_undegraded":"2024-03-24T00:35:42.425560+0000","last_fullsized":"2024-03-24T00:35:42.425560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067539+0000","last_change":"2024-03-24T00:35:42.436171+0000","last_active":"2024-03-24T00:36:41.067539+0000","last_peered":"2024-03-24T00:36:41.067539+0000","last_clean":"2024-03-24T00:36:41.067539+0000","last_became_active":"2024-03-24T00:35:42.435757+0000","last_became_peered":"2024-03-24T00:35:42.435757+0000","last_unstale":"2024-03-24T00:36:41.067539+0000","last_undegraded":"2024-03-24T00:36:41.067539+0000","last_fullsized":"2024-03-24T00:36:41.067539+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:08.194563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430666+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:42.430666+0000","last_peered":"2024-03-24T00:35:42.430666+0000","last_clean":"2024-03-24T00:35:42.430666+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:42.430666+0000","last_undegraded":"2024-03-24T00:35:42.430666+0000","last_fullsized":"2024-03-24T00:35:42.430666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059832+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:36:41.059832+0000","last_peered":"2024-03-24T00:36:41.059832+0000","last_clean":"2024-03-24T00:36:41.059832+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:36:41.059832+0000","last_undegraded":"2024-03-24T00:36:41.059832+0000","last_fullsized":"2024-03-24T00:36:41.059832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.438512+0000","last_change":"2024-03-24T00:35:42.438099+0000","last_active":"2024-03-24T00:35:42.438512+0000","last_peered":"2024-03-24T00:35:42.438512+0000","last_clean":"2024-03-24T00:35:42.438512+0000","last_became_active":"2024-03-24T00:35:42.432549+0000","last_became_peered":"2024-03-24T00:35:42.432549+0000","last_unstale":"2024-03-24T00:35:42.438512+0000","last_undegraded":"2024-03-24T00:35:42.438512+0000","last_fullsized":"2024-03-24T00:35:42.438512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:33.579265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056704+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:36:41.056704+0000","last_peered":"2024-03-24T00:36:41.056704+0000","last_clean":"2024-03-24T00:36:41.056704+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:36:41.056704+0000","last_undegraded":"2024-03-24T00:36:41.056704+0000","last_fullsized":"2024-03-24T00:36:41.056704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.434738+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:42.434738+0000","last_peered":"2024-03-24T00:35:42.434738+0000","last_clean":"2024-03-24T00:35:42.434738+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:42.434738+0000","last_undegraded":"2024-03-24T00:35:42.434738+0000","last_fullsized":"2024-03-24T00:35:42.434738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.049339+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:36:41.049339+0000","last_peered":"2024-03-24T00:36:41.049339+0000","last_clean":"2024-03-24T00:36:41.049339+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:36:41.049339+0000","last_undegraded":"2024-03-24T00:36:41.049339+0000","last_fullsized":"2024-03-24T00:36:41.049339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423562+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:42.423562+0000","last_peered":"2024-03-24T00:35:42.423562+0000","last_clean":"2024-03-24T00:35:42.423562+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:42.423562+0000","last_undegraded":"2024-03-24T00:35:42.423562+0000","last_fullsized":"2024-03-24T00:35:42.423562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082291+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:36:41.082291+0000","last_peered":"2024-03-24T00:36:41.082291+0000","last_clean":"2024-03-24T00:36:41.082291+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:36:41.082291+0000","last_undegraded":"2024-03-24T00:36:41.082291+0000","last_fullsized":"2024-03-24T00:36:41.082291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.051541+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:36:41.051541+0000","last_peered":"2024-03-24T00:36:41.051541+0000","last_clean":"2024-03-24T00:36:41.051541+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:36:41.051541+0000","last_undegraded":"2024-03-24T00:36:41.051541+0000","last_fullsized":"2024-03-24T00:36:41.051541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422997+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:42.422997+0000","last_peered":"2024-03-24T00:35:42.422997+0000","last_clean":"2024-03-24T00:35:42.422997+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:42.422997+0000","last_undegraded":"2024-03-24T00:35:42.422997+0000","last_fullsized":"2024-03-24T00:35:42.422997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064055+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:36:41.064055+0000","last_peered":"2024-03-24T00:36:41.064055+0000","last_clean":"2024-03-24T00:36:41.064055+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:36:41.064055+0000","last_undegraded":"2024-03-24T00:36:41.064055+0000","last_fullsized":"2024-03-24T00:36:41.064055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425657+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:42.425657+0000","last_peered":"2024-03-24T00:35:42.425657+0000","last_clean":"2024-03-24T00:35:42.425657+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:42.425657+0000","last_undegraded":"2024-03-24T00:35:42.425657+0000","last_fullsized":"2024-03-24T00:35:42.425657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423432+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:42.423432+0000","last_peered":"2024-03-24T00:35:42.423432+0000","last_clean":"2024-03-24T00:35:42.423432+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:42.423432+0000","last_undegraded":"2024-03-24T00:35:42.423432+0000","last_fullsized":"2024-03-24T00:35:42.423432+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060829+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:36:41.060829+0000","last_peered":"2024-03-24T00:36:41.060829+0000","last_clean":"2024-03-24T00:36:41.060829+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:36:41.060829+0000","last_undegraded":"2024-03-24T00:36:41.060829+0000","last_fullsized":"2024-03-24T00:36:41.060829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423329+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:42.423329+0000","last_peered":"2024-03-24T00:35:42.423329+0000","last_clean":"2024-03-24T00:35:42.423329+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:42.423329+0000","last_undegraded":"2024-03-24T00:35:42.423329+0000","last_fullsized":"2024-03-24T00:35:42.423329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431514+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:42.431514+0000","last_peered":"2024-03-24T00:35:42.431514+0000","last_clean":"2024-03-24T00:35:42.431514+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:42.431514+0000","last_undegraded":"2024-03-24T00:35:42.431514+0000","last_fullsized":"2024-03-24T00:35:42.431514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074782+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:36:41.074782+0000","last_peered":"2024-03-24T00:36:41.074782+0000","last_clean":"2024-03-24T00:36:41.074782+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:36:41.074782+0000","last_undegraded":"2024-03-24T00:36:41.074782+0000","last_fullsized":"2024-03-24T00:36:41.074782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:42.954936+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:37:42.954936+0000","last_peered":"2024-03-24T00:37:42.954936+0000","last_clean":"2024-03-24T00:37:42.954936+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:37:42.954936+0000","last_undegraded":"2024-03-24T00:37:42.954936+0000","last_fullsized":"2024-03-24T00:37:42.954936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066366+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:36:41.066366+0000","last_peered":"2024-03-24T00:36:41.066366+0000","last_clean":"2024-03-24T00:36:41.066366+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:36:41.066366+0000","last_undegraded":"2024-03-24T00:36:41.066366+0000","last_fullsized":"2024-03-24T00:36:41.066366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425950+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:42.425950+0000","last_peered":"2024-03-24T00:35:42.425950+0000","last_clean":"2024-03-24T00:35:42.425950+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:42.425950+0000","last_undegraded":"2024-03-24T00:35:42.425950+0000","last_fullsized":"2024-03-24T00:35:42.425950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075408+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:36:41.075408+0000","last_peered":"2024-03-24T00:36:41.075408+0000","last_clean":"2024-03-24T00:36:41.075408+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:36:41.075408+0000","last_undegraded":"2024-03-24T00:36:41.075408+0000","last_fullsized":"2024-03-24T00:36:41.075408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.862689+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:42.862689+0000","last_peered":"2024-03-24T00:35:42.862689+0000","last_clean":"2024-03-24T00:35:42.862689+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:42.862689+0000","last_undegraded":"2024-03-24T00:35:42.862689+0000","last_fullsized":"2024-03-24T00:35:42.862689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062126+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:36:41.062126+0000","last_peered":"2024-03-24T00:36:41.062126+0000","last_clean":"2024-03-24T00:36:41.062126+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:36:41.062126+0000","last_undegraded":"2024-03-24T00:36:41.062126+0000","last_fullsized":"2024-03-24T00:36:41.062126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430960+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:42.430960+0000","last_peered":"2024-03-24T00:35:42.430960+0000","last_clean":"2024-03-24T00:35:42.430960+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:42.430960+0000","last_undegraded":"2024-03-24T00:35:42.430960+0000","last_fullsized":"2024-03-24T00:35:42.430960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082706+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:36:41.082706+0000","last_peered":"2024-03-24T00:36:41.082706+0000","last_clean":"2024-03-24T00:36:41.082706+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:36:41.082706+0000","last_undegraded":"2024-03-24T00:36:41.082706+0000","last_fullsized":"2024-03-24T00:36:41.082706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424573+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:42.424573+0000","last_peered":"2024-03-24T00:35:42.424573+0000","last_clean":"2024-03-24T00:35:42.424573+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:42.424573+0000","last_undegraded":"2024-03-24T00:35:42.424573+0000","last_fullsized":"2024-03-24T00:35:42.424573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080662+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:36:41.080662+0000","last_peered":"2024-03-24T00:36:41.080662+0000","last_clean":"2024-03-24T00:36:41.080662+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:36:41.080662+0000","last_undegraded":"2024-03-24T00:36:41.080662+0000","last_fullsized":"2024-03-24T00:36:41.080662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430784+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:42.430784+0000","last_peered":"2024-03-24T00:35:42.430784+0000","last_clean":"2024-03-24T00:35:42.430784+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:42.430784+0000","last_undegraded":"2024-03-24T00:35:42.430784+0000","last_fullsized":"2024-03-24T00:35:42.430784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071515+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:36:41.071515+0000","last_peered":"2024-03-24T00:36:41.071515+0000","last_clean":"2024-03-24T00:36:41.071515+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:36:41.071515+0000","last_undegraded":"2024-03-24T00:36:41.071515+0000","last_fullsized":"2024-03-24T00:36:41.071515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.432672+0000","last_change":"2024-03-24T00:35:42.432365+0000","last_active":"2024-03-24T00:35:42.432672+0000","last_peered":"2024-03-24T00:35:42.432672+0000","last_clean":"2024-03-24T00:35:42.432672+0000","last_became_active":"2024-03-24T00:35:42.432113+0000","last_became_peered":"2024-03-24T00:35:42.432113+0000","last_unstale":"2024-03-24T00:35:42.432672+0000","last_undegraded":"2024-03-24T00:35:42.432672+0000","last_fullsized":"2024-03-24T00:35:42.432672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:32.467534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065534+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:36:41.065534+0000","last_peered":"2024-03-24T00:36:41.065534+0000","last_clean":"2024-03-24T00:36:41.065534+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:36:41.065534+0000","last_undegraded":"2024-03-24T00:36:41.065534+0000","last_fullsized":"2024-03-24T00:36:41.065534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431250+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:42.431250+0000","last_peered":"2024-03-24T00:35:42.431250+0000","last_clean":"2024-03-24T00:35:42.431250+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:42.431250+0000","last_undegraded":"2024-03-24T00:35:42.431250+0000","last_fullsized":"2024-03-24T00:35:42.431250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066395+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:36:41.066395+0000","last_peered":"2024-03-24T00:36:41.066395+0000","last_clean":"2024-03-24T00:36:41.066395+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:36:41.066395+0000","last_undegraded":"2024-03-24T00:36:41.066395+0000","last_fullsized":"2024-03-24T00:36:41.066395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.867566+0000","last_change":"2024-03-24T00:35:42.866804+0000","last_active":"2024-03-24T00:35:42.867566+0000","last_peered":"2024-03-24T00:35:42.867566+0000","last_clean":"2024-03-24T00:35:42.867566+0000","last_became_active":"2024-03-24T00:35:42.863933+0000","last_became_peered":"2024-03-24T00:35:42.863933+0000","last_unstale":"2024-03-24T00:35:42.867566+0000","last_undegraded":"2024-03-24T00:35:42.867566+0000","last_fullsized":"2024-03-24T00:35:42.867566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:55.475527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062309+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:36:41.062309+0000","last_peered":"2024-03-24T00:36:41.062309+0000","last_clean":"2024-03-24T00:36:41.062309+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:36:41.062309+0000","last_undegraded":"2024-03-24T00:36:41.062309+0000","last_fullsized":"2024-03-24T00:36:41.062309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061017+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:36:41.061017+0000","last_peered":"2024-03-24T00:36:41.061017+0000","last_clean":"2024-03-24T00:36:41.061017+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:36:41.061017+0000","last_undegraded":"2024-03-24T00:36:41.061017+0000","last_fullsized":"2024-03-24T00:36:41.061017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.053820+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:36:41.053820+0000","last_peered":"2024-03-24T00:36:41.053820+0000","last_clean":"2024-03-24T00:36:41.053820+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:36:41.053820+0000","last_undegraded":"2024-03-24T00:36:41.053820+0000","last_fullsized":"2024-03-24T00:36:41.053820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074338+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:36:41.074338+0000","last_peered":"2024-03-24T00:36:41.074338+0000","last_clean":"2024-03-24T00:36:41.074338+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:36:41.074338+0000","last_undegraded":"2024-03-24T00:36:41.074338+0000","last_fullsized":"2024-03-24T00:36:41.074338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080606+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:36:41.080606+0000","last_peered":"2024-03-24T00:36:41.080606+0000","last_clean":"2024-03-24T00:36:41.080606+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:36:41.080606+0000","last_undegraded":"2024-03-24T00:36:41.080606+0000","last_fullsized":"2024-03-24T00:36:41.080606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.076656+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:36:41.076656+0000","last_peered":"2024-03-24T00:36:41.076656+0000","last_clean":"2024-03-24T00:36:41.076656+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:36:41.076656+0000","last_undegraded":"2024-03-24T00:36:41.076656+0000","last_fullsized":"2024-03-24T00:36:41.076656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056546+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:36:41.056546+0000","last_peered":"2024-03-24T00:36:41.056546+0000","last_clean":"2024-03-24T00:36:41.056546+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:36:41.056546+0000","last_undegraded":"2024-03-24T00:36:41.056546+0000","last_fullsized":"2024-03-24T00:36:41.056546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061532+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:36:41.061532+0000","last_peered":"2024-03-24T00:36:41.061532+0000","last_clean":"2024-03-24T00:36:41.061532+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:36:41.061532+0000","last_undegraded":"2024-03-24T00:36:41.061532+0000","last_fullsized":"2024-03-24T00:36:41.061532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.069575+0000","last_change":"2024-03-24T00:35:42.437175+0000","last_active":"2024-03-24T00:36:41.069575+0000","last_peered":"2024-03-24T00:36:41.069575+0000","last_clean":"2024-03-24T00:36:41.069575+0000","last_became_active":"2024-03-24T00:35:42.436625+0000","last_became_peered":"2024-03-24T00:35:42.436625+0000","last_unstale":"2024-03-24T00:36:41.069575+0000","last_undegraded":"2024-03-24T00:36:41.069575+0000","last_fullsized":"2024-03-24T00:36:41.069575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:29:26.922046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066129+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:36:41.066129+0000","last_peered":"2024-03-24T00:36:41.066129+0000","last_clean":"2024-03-24T00:36:41.066129+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:36:41.066129+0000","last_undegraded":"2024-03-24T00:36:41.066129+0000","last_fullsized":"2024-03-24T00:36:41.066129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064052+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:36:41.064052+0000","last_peered":"2024-03-24T00:36:41.064052+0000","last_clean":"2024-03-24T00:36:41.064052+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:36:41.064052+0000","last_undegraded":"2024-03-24T00:36:41.064052+0000","last_fullsized":"2024-03-24T00:36:41.064052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064779+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:36:41.064779+0000","last_peered":"2024-03-24T00:36:41.064779+0000","last_clean":"2024-03-24T00:36:41.064779+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:36:41.064779+0000","last_undegraded":"2024-03-24T00:36:41.064779+0000","last_fullsized":"2024-03-24T00:36:41.064779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.081897+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:36:41.081897+0000","last_peered":"2024-03-24T00:36:41.081897+0000","last_clean":"2024-03-24T00:36:41.081897+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:36:41.081897+0000","last_undegraded":"2024-03-24T00:36:41.081897+0000","last_fullsized":"2024-03-24T00:36:41.081897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057088+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:36:41.057088+0000","last_peered":"2024-03-24T00:36:41.057088+0000","last_clean":"2024-03-24T00:36:41.057088+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:36:41.057088+0000","last_undegraded":"2024-03-24T00:36:41.057088+0000","last_fullsized":"2024-03-24T00:36:41.057088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68622,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":294,"num_read_kb":216,"num_write":89,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":200187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579952,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1646592,"data_stored":1124419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1056983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":528367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":528095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":466254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724619,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1056583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:56.497 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-24T00:37:56.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:37:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:56 smithi204 ceph-mon[36345]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T00:37:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:56 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1968420294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:56 smithi123 ceph-mon[31487]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T00:37:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:56 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1968420294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:56 smithi177 ceph-mon[36034]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T00:37:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:56 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1968420294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:56.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.751+0000 7ff45b400640 1 -- 172.21.15.123:0/1866982801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454104ad0 msgr2=0x7ff454104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.751+0000 7ff45b400640 1 --2- 172.21.15.123:0/1866982801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454104ad0 0x7ff454104f50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff440009ca0 tx=0x7ff44002c100 comp rx=0 tx=0).stop 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 -- 172.21.15.123:0/1866982801 shutdown_connections 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 --2- 172.21.15.123:0/1866982801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454105490 0x7ff45410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 --2- 172.21.15.123:0/1866982801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454104ad0 0x7ff454104f50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 --2- 172.21.15.123:0/1866982801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 0x7ff454103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 -- 172.21.15.123:0/1866982801 >> 172.21.15.123:0/1866982801 conn(0x7ff4540fee50 msgr2=0x7ff454101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 -- 172.21.15.123:0/1866982801 shutdown_connections 2024-03-24T00:37:56.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.752+0000 7ff45b400640 1 -- 172.21.15.123:0/1866982801 wait complete. 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 Processor -- start 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 -- start start 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 0x7ff454199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 0x7ff45419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45410f540 con 0x7ff4541038e0 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45410f3c0 con 0x7ff454105490 2024-03-24T00:37:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.753+0000 7ff45b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45410f6c0 con 0x7ff454104ad0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff459400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff459400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35786/0 (socket says 172.21.15.123:35786) 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff458a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 0x7ff454199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 0x7ff45419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff459400640 1 -- 172.21.15.123:0/700374695 learned_addr learned my addr 172.21.15.123:0/700374695 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 -- 172.21.15.123:0/700374695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 msgr2=0x7ff4541a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 -- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 msgr2=0x7ff454199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 0x7ff454199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 -- 172.21.15.123:0/700374695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4541a1850 con 0x7ff454104ad0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff459400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.754+0000 7ff453e00640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 0x7ff45419a180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff440009bf0 tx=0x7ff440035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.755+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440039040 con 0x7ff454104ad0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.755+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff440003970 con 0x7ff454104ad0 2024-03-24T00:37:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.755+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440004360 con 0x7ff454104ad0 2024-03-24T00:37:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.755+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4541a1ae0 con 0x7ff454104ad0 2024-03-24T00:37:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.755+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4541a1f50 con 0x7ff454104ad0 2024-03-24T00:37:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.756+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45410ac80 con 0x7ff454104ad0 2024-03-24T00:37:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.759+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff440046030 con 0x7ff454104ad0 2024-03-24T00:37:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.760+0000 7ff451600640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 0x7ff4380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.760+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff4400f0c40 con 0x7ff454104ad0 2024-03-24T00:37:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.760+0000 7ff458a00640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 0x7ff4380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.761+0000 7ff458a00640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 0x7ff4380a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff45419b110 tx=0x7ff444005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.762+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff44004d060 con 0x7ff454104ad0 2024-03-24T00:37:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.987+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 --> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff454061ae0 con 0x7ff4380a2800 2024-03-24T00:37:56.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:56.997+0000 7ff451600640 1 -- 172.21.15.123:0/700374695 <== mgr.14152 v2:172.21.15.123:6800/2933502234 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258757 (secure 0 0 0) 0x7ff454061ae0 con 0x7ff4380a2800 2024-03-24T00:37:57.001 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:57.003 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-24T00:37:57.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 msgr2=0x7ff4380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 0x7ff4380a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff45419b110 tx=0x7ff444005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:57.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 msgr2=0x7ff45419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 0x7ff45419a180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff440009bf0 tx=0x7ff440035d10 comp rx=0 tx=0).stop 2024-03-24T00:37:57.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 shutdown_connections 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454105490 0x7ff4541a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454104ad0 0x7ff45419a180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a2800 0x7ff4380a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 --2- 172.21.15.123:0/700374695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4541038e0 0x7ff454199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 >> 172.21.15.123:0/700374695 conn(0x7ff4540fee50 msgr2=0x7ff454107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 shutdown_connections 2024-03-24T00:37:57.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.002+0000 7ff45b400640 1 -- 172.21.15.123:0/700374695 wait complete. 2024-03-24T00:37:57.020 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-03-24T00:37:55.125464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659224,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":420,"num_read_kb":325,"num_write":272,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354108,"kb_used_data":13276,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587716,"statfs":{"total":1151940427776,"available":1151577821184,"internally_reserved":0,"allocated":13594624,"data_stored":7557612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5679,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004698"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082792+0000","last_change":"2024-03-24T00:35:32.857080+0000","last_active":"2024-03-24T00:36:41.082792+0000","last_peered":"2024-03-24T00:36:41.082792+0000","last_clean":"2024-03-24T00:36:41.082792+0000","last_became_active":"2024-03-24T00:35:32.856786+0000","last_became_peered":"2024-03-24T00:35:32.856786+0000","last_unstale":"2024-03-24T00:36:41.082792+0000","last_undegraded":"2024-03-24T00:36:41.082792+0000","last_fullsized":"2024-03-24T00:36:41.082792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:18.190081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083200+0000","last_change":"2024-03-24T00:35:32.855469+0000","last_active":"2024-03-24T00:36:41.083200+0000","last_peered":"2024-03-24T00:36:41.083200+0000","last_clean":"2024-03-24T00:36:41.083200+0000","last_became_active":"2024-03-24T00:35:32.855083+0000","last_became_peered":"2024-03-24T00:35:32.855083+0000","last_unstale":"2024-03-24T00:36:41.083200+0000","last_undegraded":"2024-03-24T00:36:41.083200+0000","last_fullsized":"2024-03-24T00:36:41.083200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:48.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082591+0000","last_change":"2024-03-24T00:35:32.853359+0000","last_active":"2024-03-24T00:36:41.082591+0000","last_peered":"2024-03-24T00:36:41.082591+0000","last_clean":"2024-03-24T00:36:41.082591+0000","last_became_active":"2024-03-24T00:35:32.852053+0000","last_became_peered":"2024-03-24T00:35:32.852053+0000","last_unstale":"2024-03-24T00:36:41.082591+0000","last_undegraded":"2024-03-24T00:36:41.082591+0000","last_fullsized":"2024-03-24T00:36:41.082591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:00.395114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059003+0000","last_change":"2024-03-24T00:35:32.839453+0000","last_active":"2024-03-24T00:36:41.059003+0000","last_peered":"2024-03-24T00:36:41.059003+0000","last_clean":"2024-03-24T00:36:41.059003+0000","last_became_active":"2024-03-24T00:35:32.838631+0000","last_became_peered":"2024-03-24T00:35:32.838631+0000","last_unstale":"2024-03-24T00:36:41.059003+0000","last_undegraded":"2024-03-24T00:36:41.059003+0000","last_fullsized":"2024-03-24T00:36:41.059003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:07.717512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.068426+0000","last_change":"2024-03-24T00:35:32.851286+0000","last_active":"2024-03-24T00:36:41.068426+0000","last_peered":"2024-03-24T00:36:41.068426+0000","last_clean":"2024-03-24T00:36:41.068426+0000","last_became_active":"2024-03-24T00:35:32.850915+0000","last_became_peered":"2024-03-24T00:35:32.850915+0000","last_unstale":"2024-03-24T00:36:41.068426+0000","last_undegraded":"2024-03-24T00:36:41.068426+0000","last_fullsized":"2024-03-24T00:36:41.068426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:10.449179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073848+0000","last_change":"2024-03-24T00:35:32.835878+0000","last_active":"2024-03-24T00:36:41.073848+0000","last_peered":"2024-03-24T00:36:41.073848+0000","last_clean":"2024-03-24T00:36:41.073848+0000","last_became_active":"2024-03-24T00:35:32.835580+0000","last_became_peered":"2024-03-24T00:35:32.835580+0000","last_unstale":"2024-03-24T00:36:41.073848+0000","last_undegraded":"2024-03-24T00:36:41.073848+0000","last_fullsized":"2024-03-24T00:36:41.073848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:39.512255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074753+0000","last_change":"2024-03-24T00:35:32.825115+0000","last_active":"2024-03-24T00:36:41.074753+0000","last_peered":"2024-03-24T00:36:41.074753+0000","last_clean":"2024-03-24T00:36:41.074753+0000","last_became_active":"2024-03-24T00:35:32.824854+0000","last_became_peered":"2024-03-24T00:35:32.824854+0000","last_unstale":"2024-03-24T00:36:41.074753+0000","last_undegraded":"2024-03-24T00:36:41.074753+0000","last_fullsized":"2024-03-24T00:36:41.074753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:33.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065233+0000","last_change":"2024-03-24T00:35:32.853155+0000","last_active":"2024-03-24T00:36:41.065233+0000","last_peered":"2024-03-24T00:36:41.065233+0000","last_clean":"2024-03-24T00:36:41.065233+0000","last_became_active":"2024-03-24T00:35:32.852449+0000","last_became_peered":"2024-03-24T00:35:32.852449+0000","last_unstale":"2024-03-24T00:36:41.065233+0000","last_undegraded":"2024-03-24T00:36:41.065233+0000","last_fullsized":"2024-03-24T00:36:41.065233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:55.947230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063340+0000","last_change":"2024-03-24T00:35:32.836904+0000","last_active":"2024-03-24T00:36:41.063340+0000","last_peered":"2024-03-24T00:36:41.063340+0000","last_clean":"2024-03-24T00:36:41.063340+0000","last_became_active":"2024-03-24T00:35:32.836600+0000","last_became_peered":"2024-03-24T00:35:32.836600+0000","last_unstale":"2024-03-24T00:36:41.063340+0000","last_undegraded":"2024-03-24T00:36:41.063340+0000","last_fullsized":"2024-03-24T00:36:41.063340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.482670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078500+0000","last_change":"2024-03-24T00:35:42.436405+0000","last_active":"2024-03-24T00:36:41.078500+0000","last_peered":"2024-03-24T00:36:41.078500+0000","last_clean":"2024-03-24T00:36:41.078500+0000","last_became_active":"2024-03-24T00:35:42.436022+0000","last_became_peered":"2024-03-24T00:35:42.436022+0000","last_unstale":"2024-03-24T00:36:41.078500+0000","last_undegraded":"2024-03-24T00:36:41.078500+0000","last_fullsized":"2024-03-24T00:36:41.078500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:59.022152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063157+0000","last_change":"2024-03-24T00:35:32.837334+0000","last_active":"2024-03-24T00:36:41.063157+0000","last_peered":"2024-03-24T00:36:41.063157+0000","last_clean":"2024-03-24T00:36:41.063157+0000","last_became_active":"2024-03-24T00:35:32.837007+0000","last_became_peered":"2024-03-24T00:35:32.837007+0000","last_unstale":"2024-03-24T00:36:41.063157+0000","last_undegraded":"2024-03-24T00:36:41.063157+0000","last_fullsized":"2024-03-24T00:36:41.063157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.289946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.046590+0000","last_change":"2024-03-24T00:35:32.841902+0000","last_active":"2024-03-24T00:36:41.046590+0000","last_peered":"2024-03-24T00:36:41.046590+0000","last_clean":"2024-03-24T00:36:41.046590+0000","last_became_active":"2024-03-24T00:35:32.841549+0000","last_became_peered":"2024-03-24T00:35:32.841549+0000","last_unstale":"2024-03-24T00:36:41.046590+0000","last_undegraded":"2024-03-24T00:36:41.046590+0000","last_fullsized":"2024-03-24T00:36:41.046590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:51.658949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.077839+0000","last_change":"2024-03-24T00:35:32.851117+0000","last_active":"2024-03-24T00:36:41.077839+0000","last_peered":"2024-03-24T00:36:41.077839+0000","last_clean":"2024-03-24T00:36:41.077839+0000","last_became_active":"2024-03-24T00:35:32.850641+0000","last_became_peered":"2024-03-24T00:35:32.850641+0000","last_unstale":"2024-03-24T00:36:41.077839+0000","last_undegraded":"2024-03-24T00:36:41.077839+0000","last_fullsized":"2024-03-24T00:36:41.077839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:58.468234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.084192+0000","last_change":"2024-03-24T00:35:32.857851+0000","last_active":"2024-03-24T00:36:41.084192+0000","last_peered":"2024-03-24T00:36:41.084192+0000","last_clean":"2024-03-24T00:36:41.084192+0000","last_became_active":"2024-03-24T00:35:32.857601+0000","last_became_peered":"2024-03-24T00:35:32.857601+0000","last_unstale":"2024-03-24T00:36:41.084192+0000","last_undegraded":"2024-03-24T00:36:41.084192+0000","last_fullsized":"2024-03-24T00:36:41.084192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.556443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083963+0000","last_change":"2024-03-24T00:35:42.433339+0000","last_active":"2024-03-24T00:36:41.083963+0000","last_peered":"2024-03-24T00:36:41.083963+0000","last_clean":"2024-03-24T00:36:41.083963+0000","last_became_active":"2024-03-24T00:35:42.432819+0000","last_became_peered":"2024-03-24T00:35:42.432819+0000","last_unstale":"2024-03-24T00:36:41.083963+0000","last_undegraded":"2024-03-24T00:36:41.083963+0000","last_fullsized":"2024-03-24T00:36:41.083963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:09.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083784+0000","last_change":"2024-03-24T00:35:32.855623+0000","last_active":"2024-03-24T00:36:41.083784+0000","last_peered":"2024-03-24T00:36:41.083784+0000","last_clean":"2024-03-24T00:36:41.083784+0000","last_became_active":"2024-03-24T00:35:32.855309+0000","last_became_peered":"2024-03-24T00:35:32.855309+0000","last_unstale":"2024-03-24T00:36:41.083784+0000","last_undegraded":"2024-03-24T00:36:41.083784+0000","last_fullsized":"2024-03-24T00:36:41.083784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:38.543515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082861+0000","last_change":"2024-03-24T00:35:32.860360+0000","last_active":"2024-03-24T00:36:41.082861+0000","last_peered":"2024-03-24T00:36:41.082861+0000","last_clean":"2024-03-24T00:36:41.082861+0000","last_became_active":"2024-03-24T00:35:32.860065+0000","last_became_peered":"2024-03-24T00:35:32.860065+0000","last_unstale":"2024-03-24T00:36:41.082861+0000","last_undegraded":"2024-03-24T00:36:41.082861+0000","last_fullsized":"2024-03-24T00:36:41.082861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:19.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.079675+0000","last_change":"2024-03-24T00:35:32.840459+0000","last_active":"2024-03-24T00:36:41.079675+0000","last_peered":"2024-03-24T00:36:41.079675+0000","last_clean":"2024-03-24T00:36:41.079675+0000","last_became_active":"2024-03-24T00:35:32.837963+0000","last_became_peered":"2024-03-24T00:35:32.837963+0000","last_unstale":"2024-03-24T00:36:41.079675+0000","last_undegraded":"2024-03-24T00:36:41.079675+0000","last_fullsized":"2024-03-24T00:36:41.079675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:08:59.671256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056950+0000","last_change":"2024-03-24T00:35:32.851298+0000","last_active":"2024-03-24T00:36:41.056950+0000","last_peered":"2024-03-24T00:36:41.056950+0000","last_clean":"2024-03-24T00:36:41.056950+0000","last_became_active":"2024-03-24T00:35:32.850998+0000","last_became_peered":"2024-03-24T00:35:32.850998+0000","last_unstale":"2024-03-24T00:36:41.056950+0000","last_undegraded":"2024-03-24T00:36:41.056950+0000","last_fullsized":"2024-03-24T00:36:41.056950+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:33.564286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435499+0000","last_change":"2024-03-24T00:35:32.345354+0000","last_active":"2024-03-24T00:35:42.435499+0000","last_peered":"2024-03-24T00:35:42.435499+0000","last_clean":"2024-03-24T00:35:42.435499+0000","last_became_active":"2024-03-24T00:35:32.344633+0000","last_became_peered":"2024-03-24T00:35:32.344633+0000","last_unstale":"2024-03-24T00:35:42.435499+0000","last_undegraded":"2024-03-24T00:35:42.435499+0000","last_fullsized":"2024-03-24T00:35:42.435499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:25.269009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.063155+0000","last_change":"2024-03-24T00:35:32.849192+0000","last_active":"2024-03-24T00:36:41.063155+0000","last_peered":"2024-03-24T00:36:41.063155+0000","last_clean":"2024-03-24T00:36:41.063155+0000","last_became_active":"2024-03-24T00:35:32.848788+0000","last_became_peered":"2024-03-24T00:35:32.848788+0000","last_unstale":"2024-03-24T00:36:41.063155+0000","last_undegraded":"2024-03-24T00:36:41.063155+0000","last_fullsized":"2024-03-24T00:36:41.063155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:47:18.495236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.433361+0000","last_change":"2024-03-24T00:35:32.361472+0000","last_active":"2024-03-24T00:35:42.433361+0000","last_peered":"2024-03-24T00:35:42.433361+0000","last_clean":"2024-03-24T00:35:42.433361+0000","last_became_active":"2024-03-24T00:35:32.360749+0000","last_became_peered":"2024-03-24T00:35:32.360749+0000","last_unstale":"2024-03-24T00:35:42.433361+0000","last_undegraded":"2024-03-24T00:35:42.433361+0000","last_fullsized":"2024-03-24T00:35:42.433361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:04.225359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080915+0000","last_change":"2024-03-24T00:35:32.854674+0000","last_active":"2024-03-24T00:36:41.080915+0000","last_peered":"2024-03-24T00:36:41.080915+0000","last_clean":"2024-03-24T00:36:41.080915+0000","last_became_active":"2024-03-24T00:35:32.852293+0000","last_became_peered":"2024-03-24T00:35:32.852293+0000","last_unstale":"2024-03-24T00:36:41.080915+0000","last_undegraded":"2024-03-24T00:36:41.080915+0000","last_fullsized":"2024-03-24T00:36:41.080915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:58.859995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.435536+0000","last_change":"2024-03-24T00:35:42.435006+0000","last_active":"2024-03-24T00:35:42.435536+0000","last_peered":"2024-03-24T00:35:42.435536+0000","last_clean":"2024-03-24T00:35:42.435536+0000","last_became_active":"2024-03-24T00:35:42.429558+0000","last_became_peered":"2024-03-24T00:35:42.429558+0000","last_unstale":"2024-03-24T00:35:42.435536+0000","last_undegraded":"2024-03-24T00:35:42.435536+0000","last_fullsized":"2024-03-24T00:35:42.435536+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:09.212462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061003+0000","last_change":"2024-03-24T00:35:32.851134+0000","last_active":"2024-03-24T00:36:41.061003+0000","last_peered":"2024-03-24T00:36:41.061003+0000","last_clean":"2024-03-24T00:36:41.061003+0000","last_became_active":"2024-03-24T00:35:32.850385+0000","last_became_peered":"2024-03-24T00:35:32.850385+0000","last_unstale":"2024-03-24T00:36:41.061003+0000","last_undegraded":"2024-03-24T00:36:41.061003+0000","last_fullsized":"2024-03-24T00:36:41.061003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:22.839815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431452+0000","last_change":"2024-03-24T00:35:32.341362+0000","last_active":"2024-03-24T00:35:42.431452+0000","last_peered":"2024-03-24T00:35:42.431452+0000","last_clean":"2024-03-24T00:35:42.431452+0000","last_became_active":"2024-03-24T00:35:32.340095+0000","last_became_peered":"2024-03-24T00:35:32.340095+0000","last_unstale":"2024-03-24T00:35:42.431452+0000","last_undegraded":"2024-03-24T00:35:42.431452+0000","last_fullsized":"2024-03-24T00:35:42.431452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:59.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060842+0000","last_change":"2024-03-24T00:35:32.838032+0000","last_active":"2024-03-24T00:36:41.060842+0000","last_peered":"2024-03-24T00:36:41.060842+0000","last_clean":"2024-03-24T00:36:41.060842+0000","last_became_active":"2024-03-24T00:35:32.837669+0000","last_became_peered":"2024-03-24T00:35:32.837669+0000","last_unstale":"2024-03-24T00:36:41.060842+0000","last_undegraded":"2024-03-24T00:36:41.060842+0000","last_fullsized":"2024-03-24T00:36:41.060842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:21.366154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431007+0000","last_change":"2024-03-24T00:35:32.338521+0000","last_active":"2024-03-24T00:35:42.431007+0000","last_peered":"2024-03-24T00:35:42.431007+0000","last_clean":"2024-03-24T00:35:42.431007+0000","last_became_active":"2024-03-24T00:35:32.338030+0000","last_became_peered":"2024-03-24T00:35:32.338030+0000","last_unstale":"2024-03-24T00:35:42.431007+0000","last_undegraded":"2024-03-24T00:35:42.431007+0000","last_fullsized":"2024-03-24T00:35:42.431007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:28.793448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074671+0000","last_change":"2024-03-24T00:35:32.840165+0000","last_active":"2024-03-24T00:36:41.074671+0000","last_peered":"2024-03-24T00:36:41.074671+0000","last_clean":"2024-03-24T00:36:41.074671+0000","last_became_active":"2024-03-24T00:35:32.837693+0000","last_became_peered":"2024-03-24T00:35:32.837693+0000","last_unstale":"2024-03-24T00:36:41.074671+0000","last_undegraded":"2024-03-24T00:36:41.074671+0000","last_fullsized":"2024-03-24T00:36:41.074671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:30.622766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422152+0000","last_change":"2024-03-24T00:35:32.344595+0000","last_active":"2024-03-24T00:35:42.422152+0000","last_peered":"2024-03-24T00:35:42.422152+0000","last_clean":"2024-03-24T00:35:42.422152+0000","last_became_active":"2024-03-24T00:35:32.344215+0000","last_became_peered":"2024-03-24T00:35:32.344215+0000","last_unstale":"2024-03-24T00:35:42.422152+0000","last_undegraded":"2024-03-24T00:35:42.422152+0000","last_fullsized":"2024-03-24T00:35:42.422152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:28.957016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061844+0000","last_change":"2024-03-24T00:35:32.849701+0000","last_active":"2024-03-24T00:36:41.061844+0000","last_peered":"2024-03-24T00:36:41.061844+0000","last_clean":"2024-03-24T00:36:41.061844+0000","last_became_active":"2024-03-24T00:35:32.849379+0000","last_became_peered":"2024-03-24T00:35:32.849379+0000","last_unstale":"2024-03-24T00:36:41.061844+0000","last_undegraded":"2024-03-24T00:36:41.061844+0000","last_fullsized":"2024-03-24T00:36:41.061844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:12.086175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422989+0000","last_change":"2024-03-24T00:35:32.352688+0000","last_active":"2024-03-24T00:35:42.422989+0000","last_peered":"2024-03-24T00:35:42.422989+0000","last_clean":"2024-03-24T00:35:42.422989+0000","last_became_active":"2024-03-24T00:35:32.347618+0000","last_became_peered":"2024-03-24T00:35:32.347618+0000","last_unstale":"2024-03-24T00:35:42.422989+0000","last_undegraded":"2024-03-24T00:35:42.422989+0000","last_fullsized":"2024-03-24T00:35:42.422989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:51.498491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071616+0000","last_change":"2024-03-24T00:35:32.849220+0000","last_active":"2024-03-24T00:36:41.071616+0000","last_peered":"2024-03-24T00:36:41.071616+0000","last_clean":"2024-03-24T00:36:41.071616+0000","last_became_active":"2024-03-24T00:35:32.848105+0000","last_became_peered":"2024-03-24T00:35:32.848105+0000","last_unstale":"2024-03-24T00:36:41.071616+0000","last_undegraded":"2024-03-24T00:36:41.071616+0000","last_fullsized":"2024-03-24T00:36:41.071616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:40.200507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430389+0000","last_change":"2024-03-24T00:35:32.340945+0000","last_active":"2024-03-24T00:35:42.430389+0000","last_peered":"2024-03-24T00:35:42.430389+0000","last_clean":"2024-03-24T00:35:42.430389+0000","last_became_active":"2024-03-24T00:35:32.340591+0000","last_became_peered":"2024-03-24T00:35:32.340591+0000","last_unstale":"2024-03-24T00:35:42.430389+0000","last_undegraded":"2024-03-24T00:35:42.430389+0000","last_fullsized":"2024-03-24T00:35:42.430389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:31.358216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.073853+0000","last_change":"2024-03-24T00:35:32.855619+0000","last_active":"2024-03-24T00:36:41.073853+0000","last_peered":"2024-03-24T00:36:41.073853+0000","last_clean":"2024-03-24T00:36:41.073853+0000","last_became_active":"2024-03-24T00:35:32.855198+0000","last_became_peered":"2024-03-24T00:35:32.855198+0000","last_unstale":"2024-03-24T00:36:41.073853+0000","last_undegraded":"2024-03-24T00:36:41.073853+0000","last_fullsized":"2024-03-24T00:36:41.073853+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:35.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422441+0000","last_change":"2024-03-24T00:35:32.342576+0000","last_active":"2024-03-24T00:35:42.422441+0000","last_peered":"2024-03-24T00:35:42.422441+0000","last_clean":"2024-03-24T00:35:42.422441+0000","last_became_active":"2024-03-24T00:35:32.342167+0000","last_became_peered":"2024-03-24T00:35:32.342167+0000","last_unstale":"2024-03-24T00:35:42.422441+0000","last_undegraded":"2024-03-24T00:35:42.422441+0000","last_fullsized":"2024-03-24T00:35:42.422441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:44.177339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078541+0000","last_change":"2024-03-24T00:35:32.844771+0000","last_active":"2024-03-24T00:36:41.078541+0000","last_peered":"2024-03-24T00:36:41.078541+0000","last_clean":"2024-03-24T00:36:41.078541+0000","last_became_active":"2024-03-24T00:35:32.844452+0000","last_became_peered":"2024-03-24T00:35:32.844452+0000","last_unstale":"2024-03-24T00:36:41.078541+0000","last_undegraded":"2024-03-24T00:36:41.078541+0000","last_fullsized":"2024-03-24T00:36:41.078541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:20.592744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423580+0000","last_change":"2024-03-24T00:35:32.341116+0000","last_active":"2024-03-24T00:35:42.423580+0000","last_peered":"2024-03-24T00:35:42.423580+0000","last_clean":"2024-03-24T00:35:42.423580+0000","last_became_active":"2024-03-24T00:35:32.340756+0000","last_became_peered":"2024-03-24T00:35:32.340756+0000","last_unstale":"2024-03-24T00:35:42.423580+0000","last_undegraded":"2024-03-24T00:35:42.423580+0000","last_fullsized":"2024-03-24T00:35:42.423580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:08.949615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.083152+0000","last_change":"2024-03-24T00:35:32.859456+0000","last_active":"2024-03-24T00:36:41.083152+0000","last_peered":"2024-03-24T00:36:41.083152+0000","last_clean":"2024-03-24T00:36:41.083152+0000","last_became_active":"2024-03-24T00:35:32.858114+0000","last_became_peered":"2024-03-24T00:35:32.858114+0000","last_unstale":"2024-03-24T00:36:41.083152+0000","last_undegraded":"2024-03-24T00:36:41.083152+0000","last_fullsized":"2024-03-24T00:36:41.083152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:32.166644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431398+0000","last_change":"2024-03-24T00:35:32.347987+0000","last_active":"2024-03-24T00:35:42.431398+0000","last_peered":"2024-03-24T00:35:42.431398+0000","last_clean":"2024-03-24T00:35:42.431398+0000","last_became_active":"2024-03-24T00:35:32.347587+0000","last_became_peered":"2024-03-24T00:35:32.347587+0000","last_unstale":"2024-03-24T00:35:42.431398+0000","last_undegraded":"2024-03-24T00:35:42.431398+0000","last_fullsized":"2024-03-24T00:35:42.431398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:14:33.071869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057836+0000","last_change":"2024-03-24T00:35:32.840418+0000","last_active":"2024-03-24T00:36:41.057836+0000","last_peered":"2024-03-24T00:36:41.057836+0000","last_clean":"2024-03-24T00:36:41.057836+0000","last_became_active":"2024-03-24T00:35:32.840099+0000","last_became_peered":"2024-03-24T00:35:32.840099+0000","last_unstale":"2024-03-24T00:36:41.057836+0000","last_undegraded":"2024-03-24T00:36:41.057836+0000","last_fullsized":"2024-03-24T00:36:41.057836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:58.601810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":341,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:54.321438+0000","last_change":"2024-03-24T00:35:32.339114+0000","last_active":"2024-03-24T00:37:54.321438+0000","last_peered":"2024-03-24T00:37:54.321438+0000","last_clean":"2024-03-24T00:37:54.321438+0000","last_became_active":"2024-03-24T00:35:32.338761+0000","last_became_peered":"2024-03-24T00:35:32.338761+0000","last_unstale":"2024-03-24T00:37:54.321438+0000","last_undegraded":"2024-03-24T00:37:54.321438+0000","last_fullsized":"2024-03-24T00:37:54.321438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:25.212156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67824,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":289,"num_read_kb":211,"num_write":32,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062432+0000","last_change":"2024-03-24T00:35:32.835407+0000","last_active":"2024-03-24T00:36:41.062432+0000","last_peered":"2024-03-24T00:36:41.062432+0000","last_clean":"2024-03-24T00:36:41.062432+0000","last_became_active":"2024-03-24T00:35:32.835146+0000","last_became_peered":"2024-03-24T00:35:32.835146+0000","last_unstale":"2024-03-24T00:36:41.062432+0000","last_undegraded":"2024-03-24T00:36:41.062432+0000","last_fullsized":"2024-03-24T00:36:41.062432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:00:30.446852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424521+0000","last_change":"2024-03-24T00:35:32.353991+0000","last_active":"2024-03-24T00:35:42.424521+0000","last_peered":"2024-03-24T00:35:42.424521+0000","last_clean":"2024-03-24T00:35:42.424521+0000","last_became_active":"2024-03-24T00:35:32.353632+0000","last_became_peered":"2024-03-24T00:35:32.353632+0000","last_unstale":"2024-03-24T00:35:42.424521+0000","last_undegraded":"2024-03-24T00:35:42.424521+0000","last_fullsized":"2024-03-24T00:35:42.424521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:56.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067885+0000","last_change":"2024-03-24T00:35:32.850908+0000","last_active":"2024-03-24T00:36:41.067885+0000","last_peered":"2024-03-24T00:36:41.067885+0000","last_clean":"2024-03-24T00:36:41.067885+0000","last_became_active":"2024-03-24T00:35:32.850464+0000","last_became_peered":"2024-03-24T00:35:32.850464+0000","last_unstale":"2024-03-24T00:36:41.067885+0000","last_undegraded":"2024-03-24T00:36:41.067885+0000","last_fullsized":"2024-03-24T00:36:41.067885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:59.688514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430861+0000","last_change":"2024-03-24T00:35:32.353520+0000","last_active":"2024-03-24T00:35:42.430861+0000","last_peered":"2024-03-24T00:35:42.430861+0000","last_clean":"2024-03-24T00:35:42.430861+0000","last_became_active":"2024-03-24T00:35:32.353222+0000","last_became_peered":"2024-03-24T00:35:32.353222+0000","last_unstale":"2024-03-24T00:35:42.430861+0000","last_undegraded":"2024-03-24T00:35:42.430861+0000","last_fullsized":"2024-03-24T00:35:42.430861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:45.083573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.078591+0000","last_change":"2024-03-24T00:35:32.848777+0000","last_active":"2024-03-24T00:36:41.078591+0000","last_peered":"2024-03-24T00:36:41.078591+0000","last_clean":"2024-03-24T00:36:41.078591+0000","last_became_active":"2024-03-24T00:35:32.848543+0000","last_became_peered":"2024-03-24T00:35:32.848543+0000","last_unstale":"2024-03-24T00:36:41.078591+0000","last_undegraded":"2024-03-24T00:36:41.078591+0000","last_fullsized":"2024-03-24T00:36:41.078591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:51.581370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425560+0000","last_change":"2024-03-24T00:35:32.337636+0000","last_active":"2024-03-24T00:35:42.425560+0000","last_peered":"2024-03-24T00:35:42.425560+0000","last_clean":"2024-03-24T00:35:42.425560+0000","last_became_active":"2024-03-24T00:35:32.337283+0000","last_became_peered":"2024-03-24T00:35:32.337283+0000","last_unstale":"2024-03-24T00:35:42.425560+0000","last_undegraded":"2024-03-24T00:35:42.425560+0000","last_fullsized":"2024-03-24T00:35:42.425560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:00:39.137948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.067539+0000","last_change":"2024-03-24T00:35:42.436171+0000","last_active":"2024-03-24T00:36:41.067539+0000","last_peered":"2024-03-24T00:36:41.067539+0000","last_clean":"2024-03-24T00:36:41.067539+0000","last_became_active":"2024-03-24T00:35:42.435757+0000","last_became_peered":"2024-03-24T00:35:42.435757+0000","last_unstale":"2024-03-24T00:36:41.067539+0000","last_undegraded":"2024-03-24T00:36:41.067539+0000","last_fullsized":"2024-03-24T00:36:41.067539+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:08.194563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430666+0000","last_change":"2024-03-24T00:35:32.349146+0000","last_active":"2024-03-24T00:35:42.430666+0000","last_peered":"2024-03-24T00:35:42.430666+0000","last_clean":"2024-03-24T00:35:42.430666+0000","last_became_active":"2024-03-24T00:35:32.348850+0000","last_became_peered":"2024-03-24T00:35:32.348850+0000","last_unstale":"2024-03-24T00:35:42.430666+0000","last_undegraded":"2024-03-24T00:35:42.430666+0000","last_fullsized":"2024-03-24T00:35:42.430666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:19.943431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.059832+0000","last_change":"2024-03-24T00:35:32.837120+0000","last_active":"2024-03-24T00:36:41.059832+0000","last_peered":"2024-03-24T00:36:41.059832+0000","last_clean":"2024-03-24T00:36:41.059832+0000","last_became_active":"2024-03-24T00:35:32.836777+0000","last_became_peered":"2024-03-24T00:35:32.836777+0000","last_unstale":"2024-03-24T00:36:41.059832+0000","last_undegraded":"2024-03-24T00:36:41.059832+0000","last_fullsized":"2024-03-24T00:36:41.059832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:09.706790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.438512+0000","last_change":"2024-03-24T00:35:42.438099+0000","last_active":"2024-03-24T00:35:42.438512+0000","last_peered":"2024-03-24T00:35:42.438512+0000","last_clean":"2024-03-24T00:35:42.438512+0000","last_became_active":"2024-03-24T00:35:42.432549+0000","last_became_peered":"2024-03-24T00:35:42.432549+0000","last_unstale":"2024-03-24T00:35:42.438512+0000","last_undegraded":"2024-03-24T00:35:42.438512+0000","last_fullsized":"2024-03-24T00:35:42.438512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:33.579265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056704+0000","last_change":"2024-03-24T00:35:32.840975+0000","last_active":"2024-03-24T00:36:41.056704+0000","last_peered":"2024-03-24T00:36:41.056704+0000","last_clean":"2024-03-24T00:36:41.056704+0000","last_became_active":"2024-03-24T00:35:32.840672+0000","last_became_peered":"2024-03-24T00:35:32.840672+0000","last_unstale":"2024-03-24T00:36:41.056704+0000","last_undegraded":"2024-03-24T00:36:41.056704+0000","last_fullsized":"2024-03-24T00:36:41.056704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:31.849845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.434738+0000","last_change":"2024-03-24T00:35:32.357807+0000","last_active":"2024-03-24T00:35:42.434738+0000","last_peered":"2024-03-24T00:35:42.434738+0000","last_clean":"2024-03-24T00:35:42.434738+0000","last_became_active":"2024-03-24T00:35:32.356573+0000","last_became_peered":"2024-03-24T00:35:32.356573+0000","last_unstale":"2024-03-24T00:35:42.434738+0000","last_undegraded":"2024-03-24T00:35:42.434738+0000","last_fullsized":"2024-03-24T00:35:42.434738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.444147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.049339+0000","last_change":"2024-03-24T00:35:32.859813+0000","last_active":"2024-03-24T00:36:41.049339+0000","last_peered":"2024-03-24T00:36:41.049339+0000","last_clean":"2024-03-24T00:36:41.049339+0000","last_became_active":"2024-03-24T00:35:32.859475+0000","last_became_peered":"2024-03-24T00:35:32.859475+0000","last_unstale":"2024-03-24T00:36:41.049339+0000","last_undegraded":"2024-03-24T00:36:41.049339+0000","last_fullsized":"2024-03-24T00:36:41.049339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:43.750228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423562+0000","last_change":"2024-03-24T00:35:32.343118+0000","last_active":"2024-03-24T00:35:42.423562+0000","last_peered":"2024-03-24T00:35:42.423562+0000","last_clean":"2024-03-24T00:35:42.423562+0000","last_became_active":"2024-03-24T00:35:32.342783+0000","last_became_peered":"2024-03-24T00:35:32.342783+0000","last_unstale":"2024-03-24T00:35:42.423562+0000","last_undegraded":"2024-03-24T00:35:42.423562+0000","last_fullsized":"2024-03-24T00:35:42.423562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:25.305359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082291+0000","last_change":"2024-03-24T00:35:32.850200+0000","last_active":"2024-03-24T00:36:41.082291+0000","last_peered":"2024-03-24T00:36:41.082291+0000","last_clean":"2024-03-24T00:36:41.082291+0000","last_became_active":"2024-03-24T00:35:32.849886+0000","last_became_peered":"2024-03-24T00:35:32.849886+0000","last_unstale":"2024-03-24T00:36:41.082291+0000","last_undegraded":"2024-03-24T00:36:41.082291+0000","last_fullsized":"2024-03-24T00:36:41.082291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:48.403254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.051541+0000","last_change":"2024-03-24T00:35:32.838747+0000","last_active":"2024-03-24T00:36:41.051541+0000","last_peered":"2024-03-24T00:36:41.051541+0000","last_clean":"2024-03-24T00:36:41.051541+0000","last_became_active":"2024-03-24T00:35:32.838435+0000","last_became_peered":"2024-03-24T00:35:32.838435+0000","last_unstale":"2024-03-24T00:36:41.051541+0000","last_undegraded":"2024-03-24T00:36:41.051541+0000","last_fullsized":"2024-03-24T00:36:41.051541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:57.481063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.422997+0000","last_change":"2024-03-24T00:35:32.357816+0000","last_active":"2024-03-24T00:35:42.422997+0000","last_peered":"2024-03-24T00:35:42.422997+0000","last_clean":"2024-03-24T00:35:42.422997+0000","last_became_active":"2024-03-24T00:35:32.357517+0000","last_became_peered":"2024-03-24T00:35:32.357517+0000","last_unstale":"2024-03-24T00:35:42.422997+0000","last_undegraded":"2024-03-24T00:35:42.422997+0000","last_fullsized":"2024-03-24T00:35:42.422997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:39.296331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064055+0000","last_change":"2024-03-24T00:35:32.859610+0000","last_active":"2024-03-24T00:36:41.064055+0000","last_peered":"2024-03-24T00:36:41.064055+0000","last_clean":"2024-03-24T00:36:41.064055+0000","last_became_active":"2024-03-24T00:35:32.859235+0000","last_became_peered":"2024-03-24T00:35:32.859235+0000","last_unstale":"2024-03-24T00:36:41.064055+0000","last_undegraded":"2024-03-24T00:36:41.064055+0000","last_fullsized":"2024-03-24T00:36:41.064055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:12:54.195601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425657+0000","last_change":"2024-03-24T00:35:32.335125+0000","last_active":"2024-03-24T00:35:42.425657+0000","last_peered":"2024-03-24T00:35:42.425657+0000","last_clean":"2024-03-24T00:35:42.425657+0000","last_became_active":"2024-03-24T00:35:32.334644+0000","last_became_peered":"2024-03-24T00:35:32.334644+0000","last_unstale":"2024-03-24T00:35:42.425657+0000","last_undegraded":"2024-03-24T00:35:42.425657+0000","last_fullsized":"2024-03-24T00:35:42.425657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:46.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423432+0000","last_change":"2024-03-24T00:34:13.101859+0000","last_active":"2024-03-24T00:35:42.423432+0000","last_peered":"2024-03-24T00:35:42.423432+0000","last_clean":"2024-03-24T00:35:42.423432+0000","last_became_active":"2024-03-24T00:34:13.101398+0000","last_became_peered":"2024-03-24T00:34:13.101398+0000","last_unstale":"2024-03-24T00:35:42.423432+0000","last_undegraded":"2024-03-24T00:35:42.423432+0000","last_fullsized":"2024-03-24T00:35:42.423432+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:25.896589+0000","last_clean_scrub_stamp":"2024-03-24T00:30:25.896589+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:51.683087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.060829+0000","last_change":"2024-03-24T00:35:32.842226+0000","last_active":"2024-03-24T00:36:41.060829+0000","last_peered":"2024-03-24T00:36:41.060829+0000","last_clean":"2024-03-24T00:36:41.060829+0000","last_became_active":"2024-03-24T00:35:32.841882+0000","last_became_peered":"2024-03-24T00:35:32.841882+0000","last_unstale":"2024-03-24T00:36:41.060829+0000","last_undegraded":"2024-03-24T00:36:41.060829+0000","last_fullsized":"2024-03-24T00:36:41.060829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:16:54.904825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.423329+0000","last_change":"2024-03-24T00:35:32.341420+0000","last_active":"2024-03-24T00:35:42.423329+0000","last_peered":"2024-03-24T00:35:42.423329+0000","last_clean":"2024-03-24T00:35:42.423329+0000","last_became_active":"2024-03-24T00:35:32.340977+0000","last_became_peered":"2024-03-24T00:35:32.340977+0000","last_unstale":"2024-03-24T00:35:42.423329+0000","last_undegraded":"2024-03-24T00:35:42.423329+0000","last_fullsized":"2024-03-24T00:35:42.423329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:22.760647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431514+0000","last_change":"2024-03-24T00:35:32.358314+0000","last_active":"2024-03-24T00:35:42.431514+0000","last_peered":"2024-03-24T00:35:42.431514+0000","last_clean":"2024-03-24T00:35:42.431514+0000","last_became_active":"2024-03-24T00:35:32.357955+0000","last_became_peered":"2024-03-24T00:35:32.357955+0000","last_unstale":"2024-03-24T00:35:42.431514+0000","last_undegraded":"2024-03-24T00:35:42.431514+0000","last_fullsized":"2024-03-24T00:35:42.431514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:40.869380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074782+0000","last_change":"2024-03-24T00:35:32.859977+0000","last_active":"2024-03-24T00:36:41.074782+0000","last_peered":"2024-03-24T00:36:41.074782+0000","last_clean":"2024-03-24T00:36:41.074782+0000","last_became_active":"2024-03-24T00:35:32.859272+0000","last_became_peered":"2024-03-24T00:35:32.859272+0000","last_unstale":"2024-03-24T00:36:41.074782+0000","last_undegraded":"2024-03-24T00:36:41.074782+0000","last_fullsized":"2024-03-24T00:36:41.074782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:11.159565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:37:42.954936+0000","last_change":"2024-03-24T00:35:32.358732+0000","last_active":"2024-03-24T00:37:42.954936+0000","last_peered":"2024-03-24T00:37:42.954936+0000","last_clean":"2024-03-24T00:37:42.954936+0000","last_became_active":"2024-03-24T00:35:32.357468+0000","last_became_peered":"2024-03-24T00:35:32.357468+0000","last_unstale":"2024-03-24T00:37:42.954936+0000","last_undegraded":"2024-03-24T00:37:42.954936+0000","last_fullsized":"2024-03-24T00:37:42.954936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:49.880878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066366+0000","last_change":"2024-03-24T00:35:32.865307+0000","last_active":"2024-03-24T00:36:41.066366+0000","last_peered":"2024-03-24T00:36:41.066366+0000","last_clean":"2024-03-24T00:36:41.066366+0000","last_became_active":"2024-03-24T00:35:32.864238+0000","last_became_peered":"2024-03-24T00:35:32.864238+0000","last_unstale":"2024-03-24T00:36:41.066366+0000","last_undegraded":"2024-03-24T00:36:41.066366+0000","last_fullsized":"2024-03-24T00:36:41.066366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:21.831266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.425950+0000","last_change":"2024-03-24T00:35:32.346302+0000","last_active":"2024-03-24T00:35:42.425950+0000","last_peered":"2024-03-24T00:35:42.425950+0000","last_clean":"2024-03-24T00:35:42.425950+0000","last_became_active":"2024-03-24T00:35:32.345790+0000","last_became_peered":"2024-03-24T00:35:32.345790+0000","last_unstale":"2024-03-24T00:35:42.425950+0000","last_undegraded":"2024-03-24T00:35:42.425950+0000","last_fullsized":"2024-03-24T00:35:42.425950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:47.081240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.075408+0000","last_change":"2024-03-24T00:35:32.861924+0000","last_active":"2024-03-24T00:36:41.075408+0000","last_peered":"2024-03-24T00:36:41.075408+0000","last_clean":"2024-03-24T00:36:41.075408+0000","last_became_active":"2024-03-24T00:35:32.861606+0000","last_became_peered":"2024-03-24T00:35:32.861606+0000","last_unstale":"2024-03-24T00:36:41.075408+0000","last_undegraded":"2024-03-24T00:36:41.075408+0000","last_fullsized":"2024-03-24T00:36:41.075408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:48.369762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.862689+0000","last_change":"2024-03-24T00:35:32.347849+0000","last_active":"2024-03-24T00:35:42.862689+0000","last_peered":"2024-03-24T00:35:42.862689+0000","last_clean":"2024-03-24T00:35:42.862689+0000","last_became_active":"2024-03-24T00:35:32.347493+0000","last_became_peered":"2024-03-24T00:35:32.347493+0000","last_unstale":"2024-03-24T00:35:42.862689+0000","last_undegraded":"2024-03-24T00:35:42.862689+0000","last_fullsized":"2024-03-24T00:35:42.862689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:45:34.693755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062126+0000","last_change":"2024-03-24T00:35:32.834851+0000","last_active":"2024-03-24T00:36:41.062126+0000","last_peered":"2024-03-24T00:36:41.062126+0000","last_clean":"2024-03-24T00:36:41.062126+0000","last_became_active":"2024-03-24T00:35:32.834290+0000","last_became_peered":"2024-03-24T00:35:32.834290+0000","last_unstale":"2024-03-24T00:36:41.062126+0000","last_undegraded":"2024-03-24T00:36:41.062126+0000","last_fullsized":"2024-03-24T00:36:41.062126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:21.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430960+0000","last_change":"2024-03-24T00:35:32.355633+0000","last_active":"2024-03-24T00:35:42.430960+0000","last_peered":"2024-03-24T00:35:42.430960+0000","last_clean":"2024-03-24T00:35:42.430960+0000","last_became_active":"2024-03-24T00:35:32.355107+0000","last_became_peered":"2024-03-24T00:35:32.355107+0000","last_unstale":"2024-03-24T00:35:42.430960+0000","last_undegraded":"2024-03-24T00:35:42.430960+0000","last_fullsized":"2024-03-24T00:35:42.430960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:43.844335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.082706+0000","last_change":"2024-03-24T00:35:32.850388+0000","last_active":"2024-03-24T00:36:41.082706+0000","last_peered":"2024-03-24T00:36:41.082706+0000","last_clean":"2024-03-24T00:36:41.082706+0000","last_became_active":"2024-03-24T00:35:32.849956+0000","last_became_peered":"2024-03-24T00:35:32.849956+0000","last_unstale":"2024-03-24T00:36:41.082706+0000","last_undegraded":"2024-03-24T00:36:41.082706+0000","last_fullsized":"2024-03-24T00:36:41.082706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:34.478079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.424573+0000","last_change":"2024-03-24T00:35:32.353296+0000","last_active":"2024-03-24T00:35:42.424573+0000","last_peered":"2024-03-24T00:35:42.424573+0000","last_clean":"2024-03-24T00:35:42.424573+0000","last_became_active":"2024-03-24T00:35:32.352830+0000","last_became_peered":"2024-03-24T00:35:32.352830+0000","last_unstale":"2024-03-24T00:35:42.424573+0000","last_undegraded":"2024-03-24T00:35:42.424573+0000","last_fullsized":"2024-03-24T00:35:42.424573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:27.832718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080662+0000","last_change":"2024-03-24T00:35:32.865396+0000","last_active":"2024-03-24T00:36:41.080662+0000","last_peered":"2024-03-24T00:36:41.080662+0000","last_clean":"2024-03-24T00:36:41.080662+0000","last_became_active":"2024-03-24T00:35:32.864921+0000","last_became_peered":"2024-03-24T00:35:32.864921+0000","last_unstale":"2024-03-24T00:36:41.080662+0000","last_undegraded":"2024-03-24T00:36:41.080662+0000","last_fullsized":"2024-03-24T00:36:41.080662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:57:36.505939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.430784+0000","last_change":"2024-03-24T00:35:32.337747+0000","last_active":"2024-03-24T00:35:42.430784+0000","last_peered":"2024-03-24T00:35:42.430784+0000","last_clean":"2024-03-24T00:35:42.430784+0000","last_became_active":"2024-03-24T00:35:32.337425+0000","last_became_peered":"2024-03-24T00:35:32.337425+0000","last_unstale":"2024-03-24T00:35:42.430784+0000","last_undegraded":"2024-03-24T00:35:42.430784+0000","last_fullsized":"2024-03-24T00:35:42.430784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:20.161466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.071515+0000","last_change":"2024-03-24T00:35:32.847941+0000","last_active":"2024-03-24T00:36:41.071515+0000","last_peered":"2024-03-24T00:36:41.071515+0000","last_clean":"2024-03-24T00:36:41.071515+0000","last_became_active":"2024-03-24T00:35:32.847608+0000","last_became_peered":"2024-03-24T00:35:32.847608+0000","last_unstale":"2024-03-24T00:36:41.071515+0000","last_undegraded":"2024-03-24T00:36:41.071515+0000","last_fullsized":"2024-03-24T00:36:41.071515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:40:46.789748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.432672+0000","last_change":"2024-03-24T00:35:42.432365+0000","last_active":"2024-03-24T00:35:42.432672+0000","last_peered":"2024-03-24T00:35:42.432672+0000","last_clean":"2024-03-24T00:35:42.432672+0000","last_became_active":"2024-03-24T00:35:42.432113+0000","last_became_peered":"2024-03-24T00:35:42.432113+0000","last_unstale":"2024-03-24T00:35:42.432672+0000","last_undegraded":"2024-03-24T00:35:42.432672+0000","last_fullsized":"2024-03-24T00:35:42.432672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:32.467534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.065534+0000","last_change":"2024-03-24T00:35:32.839371+0000","last_active":"2024-03-24T00:36:41.065534+0000","last_peered":"2024-03-24T00:36:41.065534+0000","last_clean":"2024-03-24T00:36:41.065534+0000","last_became_active":"2024-03-24T00:35:32.839060+0000","last_became_peered":"2024-03-24T00:35:32.839060+0000","last_unstale":"2024-03-24T00:36:41.065534+0000","last_undegraded":"2024-03-24T00:36:41.065534+0000","last_fullsized":"2024-03-24T00:36:41.065534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:38.064871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.431250+0000","last_change":"2024-03-24T00:35:32.354300+0000","last_active":"2024-03-24T00:35:42.431250+0000","last_peered":"2024-03-24T00:35:42.431250+0000","last_clean":"2024-03-24T00:35:42.431250+0000","last_became_active":"2024-03-24T00:35:32.353976+0000","last_became_peered":"2024-03-24T00:35:32.353976+0000","last_unstale":"2024-03-24T00:35:42.431250+0000","last_undegraded":"2024-03-24T00:35:42.431250+0000","last_fullsized":"2024-03-24T00:35:42.431250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:54.560611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066395+0000","last_change":"2024-03-24T00:35:32.859974+0000","last_active":"2024-03-24T00:36:41.066395+0000","last_peered":"2024-03-24T00:36:41.066395+0000","last_clean":"2024-03-24T00:36:41.066395+0000","last_became_active":"2024-03-24T00:35:32.859603+0000","last_became_peered":"2024-03-24T00:35:32.859603+0000","last_unstale":"2024-03-24T00:36:41.066395+0000","last_undegraded":"2024-03-24T00:36:41.066395+0000","last_fullsized":"2024-03-24T00:36:41.066395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:56.250287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:35:42.867566+0000","last_change":"2024-03-24T00:35:42.866804+0000","last_active":"2024-03-24T00:35:42.867566+0000","last_peered":"2024-03-24T00:35:42.867566+0000","last_clean":"2024-03-24T00:35:42.867566+0000","last_became_active":"2024-03-24T00:35:42.863933+0000","last_became_peered":"2024-03-24T00:35:42.863933+0000","last_unstale":"2024-03-24T00:35:42.867566+0000","last_undegraded":"2024-03-24T00:35:42.867566+0000","last_fullsized":"2024-03-24T00:35:42.867566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:31.313642+0000","last_clean_scrub_stamp":"2024-03-24T00:35:31.313642+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:55.475527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.062309+0000","last_change":"2024-03-24T00:35:32.839151+0000","last_active":"2024-03-24T00:36:41.062309+0000","last_peered":"2024-03-24T00:36:41.062309+0000","last_clean":"2024-03-24T00:36:41.062309+0000","last_became_active":"2024-03-24T00:35:32.838823+0000","last_became_peered":"2024-03-24T00:35:32.838823+0000","last_unstale":"2024-03-24T00:36:41.062309+0000","last_undegraded":"2024-03-24T00:36:41.062309+0000","last_fullsized":"2024-03-24T00:36:41.062309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:06.876828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061017+0000","last_change":"2024-03-24T00:35:32.837526+0000","last_active":"2024-03-24T00:36:41.061017+0000","last_peered":"2024-03-24T00:36:41.061017+0000","last_clean":"2024-03-24T00:36:41.061017+0000","last_became_active":"2024-03-24T00:35:32.837205+0000","last_became_peered":"2024-03-24T00:35:32.837205+0000","last_unstale":"2024-03-24T00:36:41.061017+0000","last_undegraded":"2024-03-24T00:36:41.061017+0000","last_fullsized":"2024-03-24T00:36:41.061017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:08.088119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.053820+0000","last_change":"2024-03-24T00:35:32.835830+0000","last_active":"2024-03-24T00:36:41.053820+0000","last_peered":"2024-03-24T00:36:41.053820+0000","last_clean":"2024-03-24T00:36:41.053820+0000","last_became_active":"2024-03-24T00:35:32.834473+0000","last_became_peered":"2024-03-24T00:35:32.834473+0000","last_unstale":"2024-03-24T00:36:41.053820+0000","last_undegraded":"2024-03-24T00:36:41.053820+0000","last_fullsized":"2024-03-24T00:36:41.053820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:26.210559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.074338+0000","last_change":"2024-03-24T00:35:32.851311+0000","last_active":"2024-03-24T00:36:41.074338+0000","last_peered":"2024-03-24T00:36:41.074338+0000","last_clean":"2024-03-24T00:36:41.074338+0000","last_became_active":"2024-03-24T00:35:32.850991+0000","last_became_peered":"2024-03-24T00:35:32.850991+0000","last_unstale":"2024-03-24T00:36:41.074338+0000","last_undegraded":"2024-03-24T00:36:41.074338+0000","last_fullsized":"2024-03-24T00:36:41.074338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:58.558694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.080606+0000","last_change":"2024-03-24T00:35:32.858361+0000","last_active":"2024-03-24T00:36:41.080606+0000","last_peered":"2024-03-24T00:36:41.080606+0000","last_clean":"2024-03-24T00:36:41.080606+0000","last_became_active":"2024-03-24T00:35:32.856211+0000","last_became_peered":"2024-03-24T00:35:32.856211+0000","last_unstale":"2024-03-24T00:36:41.080606+0000","last_undegraded":"2024-03-24T00:36:41.080606+0000","last_fullsized":"2024-03-24T00:36:41.080606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:43.404365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.076656+0000","last_change":"2024-03-24T00:35:32.861602+0000","last_active":"2024-03-24T00:36:41.076656+0000","last_peered":"2024-03-24T00:36:41.076656+0000","last_clean":"2024-03-24T00:36:41.076656+0000","last_became_active":"2024-03-24T00:35:32.858631+0000","last_became_peered":"2024-03-24T00:35:32.858631+0000","last_unstale":"2024-03-24T00:36:41.076656+0000","last_undegraded":"2024-03-24T00:36:41.076656+0000","last_fullsized":"2024-03-24T00:36:41.076656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:05.727615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.056546+0000","last_change":"2024-03-24T00:35:32.838463+0000","last_active":"2024-03-24T00:36:41.056546+0000","last_peered":"2024-03-24T00:36:41.056546+0000","last_clean":"2024-03-24T00:36:41.056546+0000","last_became_active":"2024-03-24T00:35:32.837969+0000","last_became_peered":"2024-03-24T00:35:32.837969+0000","last_unstale":"2024-03-24T00:36:41.056546+0000","last_undegraded":"2024-03-24T00:36:41.056546+0000","last_fullsized":"2024-03-24T00:36:41.056546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:38.593810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.061532+0000","last_change":"2024-03-24T00:35:32.836829+0000","last_active":"2024-03-24T00:36:41.061532+0000","last_peered":"2024-03-24T00:36:41.061532+0000","last_clean":"2024-03-24T00:36:41.061532+0000","last_became_active":"2024-03-24T00:35:32.836461+0000","last_became_peered":"2024-03-24T00:35:32.836461+0000","last_unstale":"2024-03-24T00:36:41.061532+0000","last_undegraded":"2024-03-24T00:36:41.061532+0000","last_fullsized":"2024-03-24T00:36:41.061532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:38.718848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.069575+0000","last_change":"2024-03-24T00:35:42.437175+0000","last_active":"2024-03-24T00:36:41.069575+0000","last_peered":"2024-03-24T00:36:41.069575+0000","last_clean":"2024-03-24T00:36:41.069575+0000","last_became_active":"2024-03-24T00:35:42.436625+0000","last_became_peered":"2024-03-24T00:35:42.436625+0000","last_unstale":"2024-03-24T00:36:41.069575+0000","last_undegraded":"2024-03-24T00:36:41.069575+0000","last_fullsized":"2024-03-24T00:36:41.069575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:29:26.922046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.066129+0000","last_change":"2024-03-24T00:35:32.851063+0000","last_active":"2024-03-24T00:36:41.066129+0000","last_peered":"2024-03-24T00:36:41.066129+0000","last_clean":"2024-03-24T00:36:41.066129+0000","last_became_active":"2024-03-24T00:35:32.850722+0000","last_became_peered":"2024-03-24T00:35:32.850722+0000","last_unstale":"2024-03-24T00:36:41.066129+0000","last_undegraded":"2024-03-24T00:36:41.066129+0000","last_fullsized":"2024-03-24T00:36:41.066129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:05.768100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064052+0000","last_change":"2024-03-24T00:35:32.856011+0000","last_active":"2024-03-24T00:36:41.064052+0000","last_peered":"2024-03-24T00:36:41.064052+0000","last_clean":"2024-03-24T00:36:41.064052+0000","last_became_active":"2024-03-24T00:35:32.855654+0000","last_became_peered":"2024-03-24T00:35:32.855654+0000","last_unstale":"2024-03-24T00:36:41.064052+0000","last_undegraded":"2024-03-24T00:36:41.064052+0000","last_fullsized":"2024-03-24T00:36:41.064052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:08.220377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.064779+0000","last_change":"2024-03-24T00:35:32.859909+0000","last_active":"2024-03-24T00:36:41.064779+0000","last_peered":"2024-03-24T00:36:41.064779+0000","last_clean":"2024-03-24T00:36:41.064779+0000","last_became_active":"2024-03-24T00:35:32.859576+0000","last_became_peered":"2024-03-24T00:35:32.859576+0000","last_unstale":"2024-03-24T00:36:41.064779+0000","last_undegraded":"2024-03-24T00:36:41.064779+0000","last_fullsized":"2024-03-24T00:36:41.064779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:45.175328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.081897+0000","last_change":"2024-03-24T00:35:32.856491+0000","last_active":"2024-03-24T00:36:41.081897+0000","last_peered":"2024-03-24T00:36:41.081897+0000","last_clean":"2024-03-24T00:36:41.081897+0000","last_became_active":"2024-03-24T00:35:32.856078+0000","last_became_peered":"2024-03-24T00:35:32.856078+0000","last_unstale":"2024-03-24T00:36:41.081897+0000","last_undegraded":"2024-03-24T00:36:41.081897+0000","last_fullsized":"2024-03-24T00:36:41.081897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:13.794727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:36:41.057088+0000","last_change":"2024-03-24T00:35:32.839008+0000","last_active":"2024-03-24T00:36:41.057088+0000","last_peered":"2024-03-24T00:36:41.057088+0000","last_clean":"2024-03-24T00:36:41.057088+0000","last_became_active":"2024-03-24T00:35:32.838692+0000","last_became_peered":"2024-03-24T00:35:32.838692+0000","last_unstale":"2024-03-24T00:36:41.057088+0000","last_undegraded":"2024-03-24T00:36:41.057088+0000","last_fullsized":"2024-03-24T00:36:41.057088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:32.315798+0000","last_clean_scrub_stamp":"2024-03-24T00:35:32.315798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:39:41.521003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68622,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":294,"num_read_kb":216,"num_write":89,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":200187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579952,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1646592,"data_stored":1124419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1056983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":528367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":528095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":466254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724619,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1056583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:57.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:57.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 -- 172.21.15.123:0/1016414142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4104ad0 msgr2=0x7f9ae4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 --2- 172.21.15.123:0/1016414142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4104ad0 0x7f9ae4104f50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009900 tx=0x7f9ad002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 -- 172.21.15.123:0/1016414142 shutdown_connections 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 --2- 172.21.15.123:0/1016414142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae4105490 0x7f9ae410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 --2- 172.21.15.123:0/1016414142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4104ad0 0x7f9ae4104f50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 --2- 172.21.15.123:0/1016414142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae41038e0 0x7f9ae4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 -- 172.21.15.123:0/1016414142 >> 172.21.15.123:0/1016414142 conn(0x7f9ae40fee90 msgr2=0x7f9ae41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 -- 172.21.15.123:0/1016414142 shutdown_connections 2024-03-24T00:37:57.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.245+0000 7f9aea800640 1 -- 172.21.15.123:0/1016414142 wait complete. 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.246+0000 7f9aea800640 1 Processor -- start 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.246+0000 7f9aea800640 1 -- start start 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae41038e0 0x7f9ae4199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 0x7f9ae41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae410f590 con 0x7f9ae4104ad0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae410f410 con 0x7f9ae4105490 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9aea800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae410f710 con 0x7f9ae41038e0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 0x7f9ae41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae41038e0 0x7f9ae4199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45488/0 (socket says 172.21.15.123:45488) 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 -- 172.21.15.123:0/3732944228 learned_addr learned my addr 172.21.15.123:0/3732944228 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 -- 172.21.15.123:0/3732944228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae41038e0 msgr2=0x7f9ae4199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae41038e0 0x7f9ae4199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 -- 172.21.15.123:0/3732944228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 msgr2=0x7f9ae41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 0x7f9ae41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae2a00640 1 -- 172.21.15.123:0/3732944228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae41a1800 con 0x7f9ae4104ad0 2024-03-24T00:37:57.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.247+0000 7f9ae3e00640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 0x7f9ae41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9ae2a00640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0000c00 tx=0x7f9ad002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0043020 con 0x7f9ae4104ad0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ad00054e0 con 0x7f9ae4104ad0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae41a1a90 con 0x7f9ae4104ad0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0003c40 con 0x7f9ae4104ad0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.248+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41a1fe0 con 0x7f9ae4104ad0 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.250+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae410ac00 con 0x7f9ae4104ad0 2024-03-24T00:37:57.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.251+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9ad0046020 con 0x7f9ae4104ad0 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.253+0000 7f9adbe00640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 0x7f9ac80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.253+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9ad00f0af0 con 0x7f9ae4104ad0 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.253+0000 7f9ae3400640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 0x7f9ac80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.254+0000 7f9ae3400640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 0x7f9ac80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4104930 tx=0x7f9ad4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.257+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ad004d020 con 0x7f9ae4104ad0 2024-03-24T00:37:57.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.462+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ae40622a0 con 0x7f9ae4104ad0 2024-03-24T00:37:57.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.464+0000 7f9adbe00640 1 -- 172.21.15.123:0/3732944228 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f9ad00bdb40 con 0x7f9ae4104ad0 2024-03-24T00:37:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:57.466 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:57.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 msgr2=0x7f9ac80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 0x7f9ac80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4104930 tx=0x7f9ad4005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:57.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 msgr2=0x7f9ae419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0000c00 tx=0x7f9ad002c650 comp rx=0 tx=0).stop 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 shutdown_connections 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ae4105490 0x7f9ae41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9ac80a2800 0x7f9ac80a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ae4104ad0 0x7f9ae419a120 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 --2- 172.21.15.123:0/3732944228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ae41038e0 0x7f9ae4199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 >> 172.21.15.123:0/3732944228 conn(0x7f9ae40fee90 msgr2=0x7f9ae4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 shutdown_connections 2024-03-24T00:37:57.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.468+0000 7f9aea800640 1 -- 172.21.15.123:0/3732944228 wait complete. 2024-03-24T00:37:57.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:57.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:57.567 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:57 smithi123 ceph-mon[31487]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:57.567 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:57 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:57.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:57 smithi204 ceph-mon[36345]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:57.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:57 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:57.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 -- 172.21.15.123:0/3597850276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a41028a0 msgr2=0x7fc3a4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3597850276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a41028a0 0x7fc3a4102d20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc390009900 tx=0x7fc39002bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 -- 172.21.15.123:0/3597850276 shutdown_connections 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3597850276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a4103260 0x7fc3a4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3597850276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a41028a0 0x7fc3a4102d20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3597850276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 -- 172.21.15.123:0/3597850276 >> 172.21.15.123:0/3597850276 conn(0x7fc3a40fcc40 msgr2=0x7fc3a40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 -- 172.21.15.123:0/3597850276 shutdown_connections 2024-03-24T00:37:57.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.690+0000 7fc3ac400640 1 -- 172.21.15.123:0/3597850276 wait complete. 2024-03-24T00:37:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.691+0000 7fc3ac400640 1 Processor -- start 2024-03-24T00:37:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.691+0000 7fc3ac400640 1 -- start start 2024-03-24T00:37:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a41028a0 0x7fc3a4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a4103260 0x7fc3a419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a410d380 con 0x7fc3a41028a0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a410d200 con 0x7fc3a4103260 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3ac400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a410d500 con 0x7fc3a41016b0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:51718/0 (socket says 172.21.15.123:51718) 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 -- 172.21.15.123:0/3668737955 learned_addr learned my addr 172.21.15.123:0/3668737955 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3aa400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a4103260 0x7fc3a419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9000640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a41028a0 0x7fc3a4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 -- 172.21.15.123:0/3668737955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a4103260 msgr2=0x7fc3a419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a4103260 0x7fc3a419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 -- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a41028a0 msgr2=0x7fc3a4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a41028a0 0x7fc3a4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.692+0000 7fc3a9a00640 1 -- 172.21.15.123:0/3668737955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a419f5e0 con 0x7fc3a41016b0 2024-03-24T00:37:57.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.693+0000 7fc3a9a00640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc39800e9c0 tx=0x7fc39800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.695+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39800cd80 con 0x7fc3a41016b0 2024-03-24T00:37:57.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.695+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3980035f0 con 0x7fc3a41016b0 2024-03-24T00:37:57.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.695+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398010690 con 0x7fc3a41016b0 2024-03-24T00:37:57.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.695+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a419f8d0 con 0x7fc3a41016b0 2024-03-24T00:37:57.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.695+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a419fdf0 con 0x7fc3a41016b0 2024-03-24T00:37:57.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.696+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4108a50 con 0x7fc3a41016b0 2024-03-24T00:37:57.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.697+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc3980037a0 con 0x7fc3a41016b0 2024-03-24T00:37:57.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.699+0000 7fc3a2000640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 0x7fc38c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.699+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc3980ced80 con 0x7fc3a41016b0 2024-03-24T00:37:57.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.699+0000 7fc3a9000640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 0x7fc38c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.700+0000 7fc3a9000640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 0x7fc38c0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc39002c240 tx=0x7fc39002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.703+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc39809bdd0 con 0x7fc3a41016b0 2024-03-24T00:37:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:57 smithi177 ceph-mon[36034]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:57 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:57.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.899+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3a40622a0 con 0x7fc3a41016b0 2024-03-24T00:37:57.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.899+0000 7fc3a2000640 1 -- 172.21.15.123:0/3668737955 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc39809b6d0 con 0x7fc3a41016b0 2024-03-24T00:37:57.900 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 msgr2=0x7fc38c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 0x7fc38c0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc39002c240 tx=0x7fc39002df90 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 msgr2=0x7fc3a41979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc39800e9c0 tx=0x7fc39800ee80 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 shutdown_connections 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3a4103260 0x7fc3a419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc38c0a2880 0x7fc38c0a4d40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3a41028a0 0x7fc3a4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 --2- 172.21.15.123:0/3668737955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3a41016b0 0x7fc3a41979c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 >> 172.21.15.123:0/3668737955 conn(0x7fc3a40fcc40 msgr2=0x7fc3a4105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 shutdown_connections 2024-03-24T00:37:57.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:57.902+0000 7fc3ac400640 1 -- 172.21.15.123:0/3668737955 wait complete. 2024-03-24T00:37:57.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 -- 172.21.15.123:0/2369043773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80100880 msgr2=0x7f8a80100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 --2- 172.21.15.123:0/2369043773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80100880 0x7f8a80100ce0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009860 tx=0x7f8a6c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 -- 172.21.15.123:0/2369043773 shutdown_connections 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 --2- 172.21.15.123:0/2369043773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a80101220 0x7f8a801016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 --2- 172.21.15.123:0/2369043773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80100880 0x7f8a80100ce0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 --2- 172.21.15.123:0/2369043773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a800fff20 0x7f8a80100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 -- 172.21.15.123:0/2369043773 >> 172.21.15.123:0/2369043773 conn(0x7f8a800fbb40 msgr2=0x7f8a800fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 -- 172.21.15.123:0/2369043773 shutdown_connections 2024-03-24T00:37:58.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.126+0000 7f8a87400640 1 -- 172.21.15.123:0/2369043773 wait complete. 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.127+0000 7f8a87400640 1 Processor -- start 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.127+0000 7f8a87400640 1 -- start start 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 0x7f8a80199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a80100880 0x7f8a8019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 0x7f8a801a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8010f520 con 0x7f8a80100880 2024-03-24T00:37:58.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8010f3a0 con 0x7f8a80101220 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a87400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8010f6a0 con 0x7f8a800fff20 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a84a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 0x7f8a80199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 0x7f8a801a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a84a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 0x7f8a80199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:51734/0 (socket says 172.21.15.123:51734) 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a84a00640 1 -- 172.21.15.123:0/123957759 learned_addr learned my addr 172.21.15.123:0/123957759 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a7fe00640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a80100880 0x7f8a8019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 -- 172.21.15.123:0/123957759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 msgr2=0x7f8a80199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 0x7f8a80199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 -- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a80100880 msgr2=0x7f8a8019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a80100880 0x7f8a8019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.128+0000 7f8a85400640 1 -- 172.21.15.123:0/123957759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a801a1810 con 0x7f8a80101220 2024-03-24T00:37:58.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a85400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 0x7f8a801a11a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009fd0 tx=0x7f8a7800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800ce20 con 0x7f8a80101220 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a78005020 con 0x7f8a80101220 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a78010720 con 0x7f8a80101220 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a801a1b00 con 0x7f8a80101220 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.129+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a801a1fd0 con 0x7f8a80101220 2024-03-24T00:37:58.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.131+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8010ae60 con 0x7f8a80101220 2024-03-24T00:37:58.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.132+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8a78005540 con 0x7f8a80101220 2024-03-24T00:37:58.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.133+0000 7f8a7d600640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 0x7f8a680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.133+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8a780ceff0 con 0x7f8a80101220 2024-03-24T00:37:58.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.133+0000 7f8a84a00640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 0x7f8a680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.134+0000 7f8a84a00640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 0x7f8a680a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8a740058c0 tx=0x7f8a74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.137+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a7809c080 con 0x7f8a80101220 2024-03-24T00:37:58.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.339+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a8002d070 con 0x7f8a80101220 2024-03-24T00:37:58.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.339+0000 7f8a7d600640 1 -- 172.21.15.123:0/123957759 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a7809b980 con 0x7f8a80101220 2024-03-24T00:37:58.340 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.341+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 msgr2=0x7f8a680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.341+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 0x7f8a680a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8a740058c0 tx=0x7f8a74009260 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.341+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 msgr2=0x7f8a801a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.341+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 0x7f8a801a11a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009fd0 tx=0x7f8a7800ef10 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 shutdown_connections 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a80101220 0x7f8a801a11a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8a680a2880 0x7f8a680a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a80100880 0x7f8a8019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 --2- 172.21.15.123:0/123957759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a800fff20 0x7f8a80199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 >> 172.21.15.123:0/123957759 conn(0x7f8a800fbb40 msgr2=0x7f8a800fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 shutdown_connections 2024-03-24T00:37:58.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.342+0000 7f8a87400640 1 -- 172.21.15.123:0/123957759 wait complete. 2024-03-24T00:37:58.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 -- 172.21.15.123:0/451771559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080104a50 msgr2=0x7f4080106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 --2- 172.21.15.123:0/451771559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080104a50 0x7f4080106e40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f406c009900 tx=0x7f406c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 -- 172.21.15.123:0/451771559 shutdown_connections 2024-03-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 --2- 172.21.15.123:0/451771559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080107380 0x7f40801097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 --2- 172.21.15.123:0/451771559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080104a50 0x7f4080106e40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 --2- 172.21.15.123:0/451771559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f4080104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 -- 172.21.15.123:0/451771559 >> 172.21.15.123:0/451771559 conn(0x7f40800fbb30 msgr2=0x7f40800fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 -- 172.21.15.123:0/451771559 shutdown_connections 2024-03-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.565+0000 7f4086e00640 1 -- 172.21.15.123:0/451771559 wait complete. 2024-03-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 Processor -- start 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 -- start start 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080104a50 0x7f4080197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080107380 0x7f408019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408010d320 con 0x7f4080107380 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408010d1a0 con 0x7f4080104a50 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.566+0000 7f4086e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408010d4a0 con 0x7f4080102120 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080104a50 0x7f4080197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:51760/0 (socket says 172.21.15.123:51760) 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 -- 172.21.15.123:0/3817696152 learned_addr learned my addr 172.21.15.123:0/3817696152 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f4084e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080107380 0x7f408019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 -- 172.21.15.123:0/3817696152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080104a50 msgr2=0x7f4080197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080104a50 0x7f4080197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 -- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080107380 msgr2=0x7f408019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080107380 0x7f408019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 -- 172.21.15.123:0/3817696152 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408019f610 con 0x7f4080102120 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.567+0000 7f407fe00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.568+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407000cd80 con 0x7f4080102120 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.568+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408019f900 con 0x7f4080102120 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.568+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4070005500 con 0x7f4080102120 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.568+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070010690 con 0x7f4080102120 2024-03-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.568+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408019fdd0 con 0x7f4080102120 2024-03-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.569+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40800fd620 con 0x7f4080102120 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3732944228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:58 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3668737955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.570+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f40700046c0 con 0x7f4080102120 2024-03-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.574+0000 7f407cc00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 0x7f40640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.574+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f40700ceea0 con 0x7f4080102120 2024-03-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.574+0000 7f407f400640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 0x7f40640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.574+0000 7f407f400640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 0x7f40640a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.576+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f407009bef0 con 0x7f4080102120 2024-03-24T00:37:58.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:37:58.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3732944228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:58.588 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.589 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.589 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:58 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3668737955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:37:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:37:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3732944228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:37:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:58.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:58 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3668737955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:37:58.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.788+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40800622a0 con 0x7f4080102120 2024-03-24T00:37:58.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.790+0000 7f407cc00640 1 -- 172.21.15.123:0/3817696152 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f407009b7f0 con 0x7f4080102120 2024-03-24T00:37:58.790 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.792+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 msgr2=0x7f40640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.792+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 0x7f40640a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.792+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 msgr2=0x7f40801979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.792+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 shutdown_connections 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f40640a2800 0x7f40640a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4080107380 0x7f408019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4080104a50 0x7f4080197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 --2- 172.21.15.123:0/3817696152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4080102120 0x7f40801979f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 >> 172.21.15.123:0/3817696152 conn(0x7f40800fbb30 msgr2=0x7f4080102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 shutdown_connections 2024-03-24T00:37:58.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:58.793+0000 7f4086e00640 1 -- 172.21.15.123:0/3817696152 wait complete. 2024-03-24T00:37:58.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 -- 172.21.15.123:0/219561412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab4102140 msgr2=0x7f3ab4104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 --2- 172.21.15.123:0/219561412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab4102140 0x7f3ab4104530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009900 tx=0x7f3aa402bc40 comp rx=0 tx=0).stop 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 -- 172.21.15.123:0/219561412 shutdown_connections 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 --2- 172.21.15.123:0/219561412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab41073a0 0x7f3ab41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 --2- 172.21.15.123:0/219561412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4104a70 0x7f3ab4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 --2- 172.21.15.123:0/219561412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab4102140 0x7f3ab4104530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 -- 172.21.15.123:0/219561412 >> 172.21.15.123:0/219561412 conn(0x7f3ab40fbb50 msgr2=0x7f3ab40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 -- 172.21.15.123:0/219561412 shutdown_connections 2024-03-24T00:37:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.002+0000 7f3ab9600640 1 -- 172.21.15.123:0/219561412 wait complete. 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 Processor -- start 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 -- start start 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab4104a70 0x7f3ab4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 0x7f3ab419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab410d3b0 con 0x7f3ab4102140 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab410d230 con 0x7f3ab4104a70 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab9600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab410d530 con 0x7f3ab41073a0 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45560/0 (socket says 172.21.15.123:45560) 2024-03-24T00:37:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3aabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab4104a70 0x7f3ab4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.003+0000 7f3ab2a00640 1 -- 172.21.15.123:0/2028611402 learned_addr learned my addr 172.21.15.123:0/2028611402 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab3400640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 0x7f3ab419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 -- 172.21.15.123:0/2028611402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 msgr2=0x7f3ab419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 0x7f3ab419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 -- 172.21.15.123:0/2028611402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab4104a70 msgr2=0x7f3ab4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab4104a70 0x7f3ab4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 -- 172.21.15.123:0/2028611402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab419f5f0 con 0x7f3ab4102140 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab3400640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 0x7f3ab419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.004+0000 7f3ab2a00640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402c8f0 tx=0x7f3aa4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.005+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4041070 con 0x7f3ab4102140 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.005+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3aa40044d0 con 0x7f3ab4102140 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.005+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab419f880 con 0x7f3ab4102140 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.005+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4034440 con 0x7f3ab4102140 2024-03-24T00:37:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.005+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab419fdc0 con 0x7f3ab4102140 2024-03-24T00:37:59.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.006+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab40fd640 con 0x7f3ab4102140 2024-03-24T00:37:59.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.008+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3aa4044070 con 0x7f3ab4102140 2024-03-24T00:37:59.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.009+0000 7f3aab400640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 0x7f3a980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.009+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3aa40f0670 con 0x7f3ab4102140 2024-03-24T00:37:59.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.009+0000 7f3aabe00640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 0x7f3a980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.010+0000 7f3aabe00640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 0x7f3a980a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4198ee0 tx=0x7f3a9c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.013+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3aa40bd6c0 con 0x7f3ab4102140 2024-03-24T00:37:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.252+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ab40622a0 con 0x7f3ab4102140 2024-03-24T00:37:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.254+0000 7f3aab400640 1 -- 172.21.15.123:0/2028611402 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f3aa40bcfc0 con 0x7f3ab4102140 2024-03-24T00:37:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:59.254 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:37:59.254 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:37:59.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.256+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 msgr2=0x7f3a980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.256+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 0x7f3a980a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4198ee0 tx=0x7f3a9c00b410 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.256+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 msgr2=0x7f3ab41979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.256+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402c8f0 tx=0x7f3aa4005d50 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 shutdown_connections 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ab41073a0 0x7f3ab419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ab4104a70 0x7f3ab4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3a980a2880 0x7f3a980a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 --2- 172.21.15.123:0/2028611402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ab4102140 0x7f3ab41979d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 >> 172.21.15.123:0/2028611402 conn(0x7f3ab40fbb50 msgr2=0x7f3ab4102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 shutdown_connections 2024-03-24T00:37:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.257+0000 7f3ab9600640 1 -- 172.21.15.123:0/2028611402 wait complete. 2024-03-24T00:37:59.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.474+0000 7fefd7c00640 1 -- 172.21.15.123:0/1290714893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0102960 msgr2=0x7fefd0102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.474+0000 7fefd7c00640 1 --2- 172.21.15.123:0/1290714893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0102960 0x7fefd0102de0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009900 tx=0x7fefbc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 -- 172.21.15.123:0/1290714893 shutdown_connections 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 --2- 172.21.15.123:0/1290714893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd0103320 0x7fefd0109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 --2- 172.21.15.123:0/1290714893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0102960 0x7fefd0102de0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 --2- 172.21.15.123:0/1290714893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd00ff990 0x7fefd00ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 -- 172.21.15.123:0/1290714893 >> 172.21.15.123:0/1290714893 conn(0x7fefd00fb280 msgr2=0x7fefd00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 -- 172.21.15.123:0/1290714893 shutdown_connections 2024-03-24T00:37:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.475+0000 7fefd7c00640 1 -- 172.21.15.123:0/1290714893 wait complete. 2024-03-24T00:37:59.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.476+0000 7fefd7c00640 1 Processor -- start 2024-03-24T00:37:59.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 -- start start 2024-03-24T00:37:59.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd00ff990 0x7fefd01022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd0102960 0x7fefd0100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd010d3c0 con 0x7fefd0102960 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd010d240 con 0x7fefd0103320 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd7c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd010d540 con 0x7fefd00ff990 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefcfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd0102960 0x7fefd0100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35874/0 (socket says 172.21.15.123:35874) 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd5c00640 1 -- 172.21.15.123:0/3933304244 learned_addr learned my addr 172.21.15.123:0/3933304244 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.477+0000 7fefd5200640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd00ff990 0x7fefd01022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 -- 172.21.15.123:0/3933304244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd00ff990 msgr2=0x7fefd01022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd00ff990 0x7fefd01022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 -- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd0102960 msgr2=0x7fefd0100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd0102960 0x7fefd0100910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 -- 172.21.15.123:0/3933304244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd01016e0 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd5c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fefc800eb10 tx=0x7fefc800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8004b20 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefc8004cc0 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd01a1af0 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.478+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8010760 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.479+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd01a2060 con 0x7fefd0103320 2024-03-24T00:37:59.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.480+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd0108a90 con 0x7fefd0103320 2024-03-24T00:37:59.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.481+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fefc8005020 con 0x7fefd0103320 2024-03-24T00:37:59.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.483+0000 7fefcd600640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 0x7fefb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.483+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fefc80cf180 con 0x7fefd0103320 2024-03-24T00:37:59.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.483+0000 7fefd5200640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 0x7fefb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.484+0000 7fefd5200640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 0x7fefb80a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fefc40095f0 tx=0x7fefc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.487+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefc809c1d0 con 0x7fefd0103320 2024-03-24T00:37:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/123957759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3817696152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:37:59 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2028611402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/123957759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3817696152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:37:59 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2028611402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:59.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.698+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefd0062340 con 0x7fefd0103320 2024-03-24T00:37:59.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.700+0000 7fefcd600640 1 -- 172.21.15.123:0/3933304244 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7fefc809bad0 con 0x7fefd0103320 2024-03-24T00:37:59.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:37:59.700 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 msgr2=0x7fefb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 0x7fefb80a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fefc40095f0 tx=0x7fefc4005f90 comp rx=0 tx=0).stop 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 msgr2=0x7fefd0100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fefc800eb10 tx=0x7fefc800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 shutdown_connections 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd0103320 0x7fefd0100e50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fefb80a2880 0x7fefb80a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefd0102960 0x7fefd0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 --2- 172.21.15.123:0/3933304244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd00ff990 0x7fefd01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 >> 172.21.15.123:0/3933304244 conn(0x7fefd00fb280 msgr2=0x7fefd0105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 shutdown_connections 2024-03-24T00:37:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.702+0000 7fefd7c00640 1 -- 172.21.15.123:0/3933304244 wait complete. 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/123957759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3817696152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:37:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:37:59 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2028611402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:37:59.721 DEBUG:tasks.fwd_scrub:Yielding 2024-03-24T00:37:59.721 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-24T00:37:59.732 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:37:59.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-24T00:37:59.733 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-24T00:37:59.733 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:37:59.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:59.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:37:59.737 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T00:37:59.749 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:37:59.749 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T00:37:59.749 INFO:tasks.workunit:timeout=3h 2024-03-24T00:37:59.749 INFO:tasks.workunit:cleanup=True 2024-03-24T00:37:59.749 DEBUG:teuthology.orchestra.run.smithi123:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-24 00:35:39.399022789 +0000 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-24 00:35:39.399022789 +0000 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-24 00:37:51.111620053 +0000 2024-03-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-24 00:35:39.399022789 +0000 2024-03-24T00:37:59.791 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:59.791 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T00:37:59.914 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 -- 172.21.15.123:0/315355881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918102120 msgr2=0x7fd918104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 --2- 172.21.15.123:0/315355881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918102120 0x7fd918104510 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd908009900 tx=0x7fd90802bc30 comp rx=0 tx=0).stop 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 -- 172.21.15.123:0/315355881 shutdown_connections 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 --2- 172.21.15.123:0/315355881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918107380 0x7fd9181097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 --2- 172.21.15.123:0/315355881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918104a50 0x7fd918106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 --2- 172.21.15.123:0/315355881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918102120 0x7fd918104510 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 -- 172.21.15.123:0/315355881 >> 172.21.15.123:0/315355881 conn(0x7fd9180fbb30 msgr2=0x7fd9180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 -- 172.21.15.123:0/315355881 shutdown_connections 2024-03-24T00:37:59.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.937+0000 7fd91ca00640 1 -- 172.21.15.123:0/315355881 wait complete. 2024-03-24T00:37:59.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.938+0000 7fd91ca00640 1 Processor -- start 2024-03-24T00:37:59.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.938+0000 7fd91ca00640 1 -- start start 2024-03-24T00:37:59.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 0x7fd918101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918107380 0x7fd918100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91810d380 con 0x7fd918102120 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91810d200 con 0x7fd918104a50 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd91ca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91810d500 con 0x7fd918107380 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 0x7fd918101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35890/0 (socket says 172.21.15.123:35890) 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 -- 172.21.15.123:0/576933641 learned_addr learned my addr 172.21.15.123:0/576933641 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 0x7fd918101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45584/0 (socket says 172.21.15.123:45584) 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd916800640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918107380 0x7fd918100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 -- 172.21.15.123:0/576933641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918107380 msgr2=0x7fd918100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918107380 0x7fd918100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 -- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 msgr2=0x7fd918101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 0x7fd918101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.939+0000 7fd915400640 1 -- 172.21.15.123:0/576933641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd918100ea0 con 0x7fd918104a50 2024-03-24T00:37:59.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd915400640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd90400e9a0 tx=0x7fd90400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd90ea00640 1 -- 172.21.15.123:0/576933641 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400cd80 con 0x7fd918104a50 2024-03-24T00:37:59.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd90ea00640 1 -- 172.21.15.123:0/576933641 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9040035f0 con 0x7fd918104a50 2024-03-24T00:37:59.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd918071cf0 con 0x7fd918104a50 2024-03-24T00:37:59.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd90ea00640 1 -- 172.21.15.123:0/576933641 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd904010670 con 0x7fd918104a50 2024-03-24T00:37:59.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.940+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd918072290 con 0x7fd918104a50 2024-03-24T00:37:59.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.944+0000 7fd90ea00640 1 -- 172.21.15.123:0/576933641 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd9040037a0 con 0x7fd918104a50 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.944+0000 7fd91f79d740 1 Processor -- start 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.944+0000 7fd91f79d740 1 -- start start 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a204590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a207090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a164eb0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a10da60 con 0x559c3a204c00 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd91f79d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a160f50 con 0x559c3a1ffa90 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd915400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a207090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd915e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a204590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45594/0 (socket says 172.21.15.123:45594) 2024-03-24T00:37:59.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 -- 172.21.15.123:0/2103238387 learned_addr learned my addr 172.21.15.123:0/2103238387 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 -- 172.21.15.123:0/2103238387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 msgr2=0x559c3a204590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a204590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 -- 172.21.15.123:0/2103238387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 msgr2=0x559c3a207090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a207090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd90ea00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 0x7fd8fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.945+0000 7fd916800640 1 -- 172.21.15.123:0/2103238387 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c39f417e0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.946+0000 7fd90ea00640 1 -- 172.21.15.123:0/576933641 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd9040cec90 con 0x7fd918104a50 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.946+0000 7fd916800640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 0x7fd8fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.946+0000 7fd916800640 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd910009fd0 tx=0x7fd9100118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a2ee757721c488 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.946+0000 7fd903000640 1 -- 172.21.15.123:0/2103238387 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910004250 con 0x559c3a1ff0a0 2024-03-24T00:37:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.947+0000 7fd903000640 1 -- 172.21.15.123:0/2103238387 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd910003db0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.947+0000 7fd903000640 1 -- 172.21.15.123:0/2103238387 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91001bc20 con 0x559c3a1ff0a0 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- 172.21.15.123:0/2103238387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 msgr2=0x559c3a1ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd910009fd0 tx=0x7fd9100118d0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd916800640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 0x7fd8fc0a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd910020000 tx=0x7fd910021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- 172.21.15.123:0/2103238387 shutdown_connections 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a207090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a204590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2103238387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ff0a0 0x559c3a1ff4c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- 172.21.15.123:0/2103238387 >> 172.21.15.123:0/2103238387 conn(0x559c3a1fa610 msgr2=0x559c3a1faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- 172.21.15.123:0/2103238387 shutdown_connections 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- 172.21.15.123:0/2103238387 wait complete. 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.948+0000 7fd91f79d740 1 -- start start 2024-03-24T00:37:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 Processor -- start 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a2097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 0x559c3a210800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a207bb0 con 0x559c3a1ffa90 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a120eb0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3a207d30 con 0x559c3a204c00 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd91f79d740 10 client.0 _ll_register_callbacks cb 0x559c3a115150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd915e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd915e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35914/0 (socket says 172.21.15.123:35914) 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd915e00640 1 -- 172.21.15.123:0/3234204358 learned_addr learned my addr 172.21.15.123:0/3234204358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.949+0000 7fd915400640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a2097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 -- 172.21.15.123:0/3234204358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 msgr2=0x559c3a210800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd916800640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 0x559c3a210800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 0x559c3a210800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 -- 172.21.15.123:0/3234204358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 msgr2=0x559c3a2097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a2097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 -- 172.21.15.123:0/3234204358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c39fa1f70 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd916800640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 0x559c3a210800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd915e00640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd908005ab0 tx=0x7fd908004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e3e2c0f6e8601d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd8f7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908005d50 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd9080046c0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.950+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd910003db0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9080048c0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 10 client.25232 fetch_fsmap learned FSMap version 19 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559c39e39170 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd910003db0 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd908003680 con 0x559c3a1ff0a0 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 10 client.25232 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:37:59.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 10 client.25232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 20 client.25232 populate_metadata read hostname 'smithi123' 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a2174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 4 client.25232 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.951+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c3a1723f0 con 0x559c3a215060 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.952+0000 7fd915400640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a2174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.952+0000 7fd915400640 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a2174d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=320586e5c9a16718 server_cookie=5c45804f3cc12aba in_seq=0 out_seq=0 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.952+0000 7fd8f7400640 10 client.25232 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd8f7400640 1 -- 172.21.15.123:0/3234204358 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559c3a1723f0 con 0x559c3a215060 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd8f7400640 10 client.25232 handle_command_reply: tid=0 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 client.25232 shutdown 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 msgr2=0x559c3a209280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd908005ab0 tx=0x7fd908004270 comp rx=0 tx=0).stop 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 shutdown_connections 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a2174d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a204c00 0x559c3a210800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a2097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd8f7400640 0 client.25232 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 --2- 172.21.15.123:0/3234204358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a209280 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 >> 172.21.15.123:0/3234204358 conn(0x559c3a1fa610 msgr2=0x559c3a1f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 shutdown_connections 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 1 -- 172.21.15.123:0/3234204358 wait complete. 2024-03-24T00:37:59.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.953+0000 7fd91f79d740 20 client.25232 trim_cache size 0 max 16384 2024-03-24T00:37:59.962 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T00:37:59.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.975+0000 7fd91f79d740 1 Processor -- start 2024-03-24T00:37:59.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 -- start start 2024-03-24T00:38:00.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a1f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a21bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b1540 con 0x559c3a1ffa90 2024-03-24T00:38:00.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b3b00 con 0x559c3a204c00 2024-03-24T00:38:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd91f79d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b1b40 con 0x559c3a1ff0a0 2024-03-24T00:38:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd916800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a21bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd915e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd915400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a1f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd915e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:51842/0 (socket says 172.21.15.123:51842) 2024-03-24T00:38:00.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd915e00640 1 -- 172.21.15.123:0/2810385393 learned_addr learned my addr 172.21.15.123:0/2810385393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd916800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a21bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35916/0 (socket says 172.21.15.123:35916) 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.976+0000 7fd915400640 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a1f9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45604/0 (socket says 172.21.15.123:45604) 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 -- 172.21.15.123:0/2810385393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 msgr2=0x559c3a21bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a21bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 -- 172.21.15.123:0/2810385393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 msgr2=0x559c3a1f9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a1f9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 -- 172.21.15.123:0/2810385393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c3a1fec60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd915e00640 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd9080039f0 tx=0x7fd90802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13019dced238dffc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd902600640 1 -- 172.21.15.123:0/2810385393 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908000e60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd902600640 1 -- 172.21.15.123:0/2810385393 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd908003c60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.977+0000 7fd902600640 1 -- 172.21.15.123:0/2810385393 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90805a070 con 0x559c3a1ff0a0 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.978+0000 7fd91f79d740 1 -- 172.21.15.123:0/2810385393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 msgr2=0x559c3a1f96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.978+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd9080039f0 tx=0x7fd90802bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.978+0000 7fd91f79d740 1 -- 172.21.15.123:0/2810385393 shutdown_connections 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.978+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a204c00 0x559c3a21bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.978+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a1ffa90 0x559c3a1f9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 --2- 172.21.15.123:0/2810385393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ff0a0 0x559c3a1f96b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 -- 172.21.15.123:0/2810385393 >> 172.21.15.123:0/2810385393 conn(0x559c3a1fa610 msgr2=0x559c3a1f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 -- 172.21.15.123:0/2810385393 shutdown_connections 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 -- 172.21.15.123:0/2810385393 wait complete. 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 -- start start 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.979+0000 7fd91f79d740 1 Processor -- start 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a20d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a204c00 0x559c3a20de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b2180 con 0x559c3a204c00 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b34a0 con 0x559c3a1ff0a0 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181b27e0 con 0x559c3a1ffa90 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd91f79d740 10 client.0 _ll_register_callbacks cb 0x7fd908009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd915e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd915e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35922/0 (socket says 172.21.15.123:35922) 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd915e00640 1 -- 172.21.15.123:0/77095193 learned_addr learned my addr 172.21.15.123:0/77095193 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd916800640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a204c00 0x559c3a20de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.980+0000 7fd915400640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a20d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 -- 172.21.15.123:0/77095193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 msgr2=0x559c3a20d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a20d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 -- 172.21.15.123:0/77095193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a204c00 msgr2=0x559c3a20de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a204c00 0x559c3a20de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 -- 172.21.15.123:0/77095193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c3a20e390 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd915e00640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd9080039c0 tx=0x7fd9080050e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2ac5c2a38ffdcc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd8f7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908000e60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd908005460 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.981+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd908003c60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd90805a040 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 10 client.25238 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559c3a271670 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd908003c60 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd90804b960 con 0x559c3a1ff0a0 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 10 client.25238 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 10 client.25238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:00.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 20 client.25238 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a273f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 4 client.25238 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.982+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559c3a16c740 con 0x559c3a215060 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.983+0000 7fd915400640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a273f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.983+0000 7fd915400640 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a273f50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d6a335cd8fddce5 server_cookie=391b8f489c4a83fb in_seq=0 out_seq=0 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.983+0000 7fd8f7400640 10 client.25238 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd8f7400640 1 -- 172.21.15.123:0/77095193 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559c3a16c740 con 0x559c3a215060 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd8f7400640 10 client.25238 handle_command_reply: tid=0 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 client.25238 shutdown 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 msgr2=0x559c3a20f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd9080039c0 tx=0x7fd9080050e0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 shutdown_connections 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c3a204c00 0x559c3a20de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x559c3a215060 0x559c3a273f50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559c3a1ffa90 0x559c3a20d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 --2- 172.21.15.123:0/77095193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3a1ff0a0 0x559c3a20f2b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd8f7400640 0 client.25238 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 >> 172.21.15.123:0/77095193 conn(0x559c3a1fa610 msgr2=0x559c3a1f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.984+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 shutdown_connections 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91f79d740 1 -- 172.21.15.123:0/77095193 wait complete. 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91f79d740 20 client.25238 trim_cache size 0 max 16384 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "d89bec1e-632e-4412-af45-7f9e99071cf2", 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:38:00.011 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 msgr2=0x7fd8fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 0x7fd8fc0a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd910020000 tx=0x7fd910021000 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 msgr2=0x7fd9181000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.985+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd90400e9a0 tx=0x7fd90400ee60 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 shutdown_connections 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd918107380 0x7fd918100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd918104a50 0x7fd9181000d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd8fc0a2880 0x7fd8fc0a4d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 --2- 172.21.15.123:0/576933641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd918102120 0x7fd918101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 >> 172.21.15.123:0/576933641 conn(0x7fd9180fbb30 msgr2=0x7fd918102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 shutdown_connections 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:37:59.986+0000 7fd91ca00640 1 -- 172.21.15.123:0/576933641 wait complete. 2024-03-24T00:38:00.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:38:00.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:38:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 -- 172.21.15.123:0/1146835132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc102140 msgr2=0x7fa9fc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 --2- 172.21.15.123:0/1146835132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc102140 0x7fa9fc104530 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec009900 tx=0x7fa9ec02bc50 comp rx=0 tx=0).stop 2024-03-24T00:38:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 -- 172.21.15.123:0/1146835132 shutdown_connections 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 --2- 172.21.15.123:0/1146835132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 --2- 172.21.15.123:0/1146835132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc104a70 0x7fa9fc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 --2- 172.21.15.123:0/1146835132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc102140 0x7fa9fc104530 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 -- 172.21.15.123:0/1146835132 >> 172.21.15.123:0/1146835132 conn(0x7fa9fc0fbb30 msgr2=0x7fa9fc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 -- 172.21.15.123:0/1146835132 shutdown_connections 2024-03-24T00:38:00.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.273+0000 7faa03400640 1 -- 172.21.15.123:0/1146835132 wait complete. 2024-03-24T00:38:00.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.274+0000 7faa03400640 1 Processor -- start 2024-03-24T00:38:00.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.274+0000 7faa03400640 1 -- start start 2024-03-24T00:38:00.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc102140 0x7fa9fc101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc10d3b0 con 0x7fa9fc1073a0 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc10d230 con 0x7fa9fc104a70 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa03400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc10d530 con 0x7fa9fc102140 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa01400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35936/0 (socket says 172.21.15.123:35936) 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa00a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc102140 0x7fa9fc101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 -- 172.21.15.123:0/4049749432 learned_addr learned my addr 172.21.15.123:0/4049749432 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 -- 172.21.15.123:0/4049749432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc102140 msgr2=0x7fa9fc101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc102140 0x7fa9fc101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 -- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 msgr2=0x7fa9fc100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7fa9f3e00640 1 -- 172.21.15.123:0/4049749432 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc100ec0 con 0x7fa9fc104a70 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.275+0000 7faa01400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e9c0 tx=0x7fa9e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7fa9fa000640 1 -- 172.21.15.123:0/4049749432 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400cd80 con 0x7fa9fc104a70 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7fa9fa000640 1 -- 172.21.15.123:0/4049749432 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e4005500 con 0x7fa9fc104a70 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7fa9fa000640 1 -- 172.21.15.123:0/4049749432 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4010690 con 0x7fa9fc104a70 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc1a1b70 con 0x7fa9fc104a70 2024-03-24T00:38:00.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.276+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc1a2110 con 0x7fa9fc104a70 2024-03-24T00:38:00.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.279+0000 7fa9fa000640 1 -- 172.21.15.123:0/4049749432 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa9e4010830 con 0x7fa9fc104a70 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.279+0000 7faa06020740 1 Processor -- start 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 -- start start 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac7052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac70ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac66a250 con 0x5618ac704e80 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac612a30 con 0x5618ac705870 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa06020740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac6662f0 con 0x5618ac70a9e0 2024-03-24T00:38:00.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa00a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa00a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35946/0 (socket says 172.21.15.123:35946) 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7faa00a00640 1 -- 172.21.15.123:0/3553948674 learned_addr learned my addr 172.21.15.123:0/3553948674 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.280+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac70ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 -- 172.21.15.123:0/3553948674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 msgr2=0x5618ac70ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac70ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 -- 172.21.15.123:0/3553948674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 msgr2=0x5618ac7052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa01400640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac7052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac7052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 -- 172.21.15.123:0/3553948674 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618ac5fd690 con 0x5618ac705870 2024-03-24T00:38:00.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa01400640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac7052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7faa00a00640 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02c900 tx=0x7fa9ec009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf10e16ffe40df43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7fa9fa000640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 0x7fa9e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7fa9f1c00640 1 -- 172.21.15.123:0/3553948674 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec042070 con 0x5618ac705870 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7fa9f1c00640 1 -- 172.21.15.123:0/3553948674 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9ec004040 con 0x5618ac705870 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.281+0000 7fa9f1c00640 1 -- 172.21.15.123:0/3553948674 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec004ed0 con 0x5618ac705870 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7fa9fa000640 1 -- 172.21.15.123:0/4049749432 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa9e40cee30 con 0x7fa9fc104a70 2024-03-24T00:38:00.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa01400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 0x7fa9e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 -- 172.21.15.123:0/3553948674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 msgr2=0x5618ac70a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02c900 tx=0x7fa9ec009670 comp rx=0 tx=0).stop 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa01400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 0x7fa9e00a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009fd0 tx=0x7fa9f4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 -- 172.21.15.123:0/3553948674 shutdown_connections 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac70ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac70a370 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 --2- 172.21.15.123:0/3553948674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac7052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.282+0000 7faa06020740 1 -- 172.21.15.123:0/3553948674 >> 172.21.15.123:0/3553948674 conn(0x5618ac6cd200 msgr2=0x5618ac7007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 -- 172.21.15.123:0/3553948674 shutdown_connections 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 -- 172.21.15.123:0/3553948674 wait complete. 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 -- start start 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 Processor -- start 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac70ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac705870 0x5618ac70f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.283+0000 7faa06020740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac70d930 con 0x5618ac705870 2024-03-24T00:38:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa06020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac625e80 con 0x5618ac70a9e0 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa06020740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ac70dab0 con 0x5618ac704e80 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa06020740 10 client.0 _ll_register_callbacks cb 0x5618ac638130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac705870 0x5618ac70f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa00a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac70ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35948/0 (socket says 172.21.15.123:35948) 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 -- 172.21.15.123:0/3324951671 learned_addr learned my addr 172.21.15.123:0/3324951671 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 -- 172.21.15.123:0/3324951671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 msgr2=0x5618ac70ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac70ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 -- 172.21.15.123:0/3324951671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac705870 msgr2=0x5618ac70f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac705870 0x5618ac70f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.284+0000 7faa01400640 1 -- 172.21.15.123:0/3324951671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618ac4487d0 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7faa01400640 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4003bf0 tx=0x7fa9f4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d727cc020ee96bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7fa9ea000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4028400 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa9f40285a0 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9ec004040 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.285+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9f40287a0 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 10 client.25250 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5618ac37e3f0 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9ec004040 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa9f4031020 con 0x5618ac70a9e0 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 10 client.25250 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 10 client.25250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 20 client.25250 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac71ac30 0x5618ac71d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 4 client.25250 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618ac677790 con 0x5618ac71ac30 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.286+0000 7faa00a00640 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac71ac30 0x5618ac71d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.287+0000 7fa9ea000640 10 client.25250 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.287+0000 7faa00a00640 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac71ac30 0x5618ac71d0d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7428d6b679d9c55 server_cookie=c5aecbb016c54c23 in_seq=0 out_seq=0 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7fa9ea000640 1 -- 172.21.15.123:0/3324951671 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5618ac677790 con 0x5618ac71ac30 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7fa9ea000640 10 client.25250 handle_command_reply: tid=0 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 client.25250 shutdown 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 msgr2=0x5618ac7163b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4003bf0 tx=0x7fa9f4004a40 comp rx=0 tx=0).stop 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 shutdown_connections 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac70a9e0 0x5618ac7163b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac705870 0x5618ac70f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac71ac30 0x5618ac71d0d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 --2- 172.21.15.123:0/3324951671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac70ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 >> 172.21.15.123:0/3324951671 conn(0x5618ac6cd200 msgr2=0x5618ac6ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7fa9ea000640 0 client.25250 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 shutdown_connections 2024-03-24T00:38:00.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 1 -- 172.21.15.123:0/3324951671 wait complete. 2024-03-24T00:38:00.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.288+0000 7faa06020740 20 client.25250 trim_cache size 0 max 16384 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.305+0000 7faa06020740 1 Processor -- start 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.305+0000 7faa06020740 1 -- start start 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac709e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac703a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1a8dc0 con 0x5618ac704e80 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1ab380 con 0x5618ac705870 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa06020740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1a93c0 con 0x5618ac70a9e0 2024-03-24T00:38:00.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35960/0 (socket says 172.21.15.123:35960) 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3492770120 learned_addr learned my addr 172.21.15.123:0/3492770120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa00a00640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac709e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa01400640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac703a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3492770120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 msgr2=0x5618ac703a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac703a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3492770120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 msgr2=0x5618ac709e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac709e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3492770120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618ac33e0f0 con 0x5618ac705870 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.306+0000 7faa00a00640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac709e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:00.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.307+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40cc940 tx=0x7fa9e4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a433c56a74ff45e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.307+0000 7fa9f1200640 1 -- 172.21.15.123:0/3492770120 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e409b320 con 0x5618ac705870 2024-03-24T00:38:00.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.307+0000 7fa9f1200640 1 -- 172.21.15.123:0/3492770120 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e409bee0 con 0x5618ac705870 2024-03-24T00:38:00.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.307+0000 7fa9f1200640 1 -- 172.21.15.123:0/3492770120 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e409b140 con 0x5618ac705870 2024-03-24T00:38:00.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- 172.21.15.123:0/3492770120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 msgr2=0x5618ac716370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40cc940 tx=0x7fa9e4005880 comp rx=0 tx=0).stop 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- 172.21.15.123:0/3492770120 shutdown_connections 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac70a9e0 0x5618ac703a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716370 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 --2- 172.21.15.123:0/3492770120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac704e80 0x5618ac709e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- 172.21.15.123:0/3492770120 >> 172.21.15.123:0/3492770120 conn(0x5618ac6cd200 msgr2=0x5618ac706ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- 172.21.15.123:0/3492770120 shutdown_connections 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- 172.21.15.123:0/3492770120 wait complete. 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.308+0000 7faa06020740 1 -- start start 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 Processor -- start 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac715b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac70a9e0 0x5618ac714140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1a9a00 con 0x5618ac70a9e0 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1aad20 con 0x5618ac705870 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1aa060 con 0x5618ac704e80 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa06020740 10 client.0 _ll_register_callbacks cb 0x7fa9f4011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7faa01400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac70a9e0 0x5618ac714140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7fa9f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35972/0 (socket says 172.21.15.123:35972) 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.309+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3407085056 learned_addr learned my addr 172.21.15.123:0/3407085056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7faa00a00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac715b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3407085056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 msgr2=0x5618ac715b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac715b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3407085056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac70a9e0 msgr2=0x5618ac714140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac70a9e0 0x5618ac714140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 -- 172.21.15.123:0/3407085056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618ac714780 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9f3e00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40cccb0 tx=0x7fa9e4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b91d759515b20724 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9ea000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40ba060 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa9e4097e50 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9e409bee0 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.310+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9e409b320 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 10 client.25256 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5618ac776f00 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e409bee0 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa9e40a3af0 con 0x5618ac705870 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 10 client.25256 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 10 client.25256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:00.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 20 client.25256 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac7777c0 0x5618ac779be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 4 client.25256 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5618ac671ae0 con 0x5618ac7777c0 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.311+0000 7faa00a00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac7777c0 0x5618ac779be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa00a00640 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac7777c0 0x5618ac779be0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fabf60c669e49b9 server_cookie=d5ca0c0830933fbe in_seq=0 out_seq=0 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7fa9ea000640 10 client.25256 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7fa9ea000640 1 -- 172.21.15.123:0/3407085056 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5618ac671ae0 con 0x5618ac7777c0 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7fa9ea000640 10 client.25256 handle_command_reply: tid=0 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 client.25256 shutdown 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 msgr2=0x5618ac716070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40cccb0 tx=0x7fa9e4009700 comp rx=0 tx=0).stop 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 shutdown_connections 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5618ac70a9e0 0x5618ac714140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618ac705870 0x5618ac716070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5618ac7777c0 0x5618ac779be0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 --2- 172.21.15.123:0/3407085056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5618ac704e80 0x5618ac715b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 >> 172.21.15.123:0/3407085056 conn(0x5618ac6cd200 msgr2=0x5618ac706ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7fa9ea000640 0 client.25256 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 shutdown_connections 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 1 -- 172.21.15.123:0/3407085056 wait complete. 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.312+0000 7faa06020740 20 client.25256 trim_cache size 0 max 16384 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:38:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: "d89bec1e-632e-4412-af45-7f9e99071cf2": { 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "d89bec1e-632e-4412-af45-7f9e99071cf2", 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:38:00.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 msgr2=0x7fa9e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 0x7fa9e00a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009fd0 tx=0x7fa9f4009500 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 msgr2=0x7fa9fc1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e9c0 tx=0x7fa9e400ee80 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 shutdown_connections 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa9e00a2880 0x7fa9e00a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9fc1073a0 0x7fa9fc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9fc104a70 0x7fa9fc1000f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 --2- 172.21.15.123:0/4049749432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9fc102140 0x7fa9fc101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.313+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 >> 172.21.15.123:0/4049749432 conn(0x7fa9fc0fbb30 msgr2=0x7fa9fc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.314+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 shutdown_connections 2024-03-24T00:38:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:00.314+0000 7faa03400640 1 -- 172.21.15.123:0/4049749432 wait complete. 2024-03-24T00:38:00.335 INFO:tasks.cephfs.filesystem:scrub status for tag:d89bec1e-632e-4412-af45-7f9e99071cf2 - {'path': '/', 'tag': 'd89bec1e-632e-4412-af45-7f9e99071cf2', 'options': 'recursive,force'} 2024-03-24T00:38:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:38:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3933304244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:38:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:38:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:00.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:38:00.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:00 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:38:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3933304244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:38:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:38:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:38:00.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:00 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T00:38:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3933304244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:38:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:38:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:38:00.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:00 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:01 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:01 smithi123 ceph-mon[31487]: pgmap v389: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:01 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:01 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:01 smithi177 ceph-mon[36034]: pgmap v389: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:01 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:01 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:01 smithi204 ceph-mon[36345]: pgmap v389: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:02 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:38:02.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:02.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:02.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:02.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:02 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:38:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:38:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:02 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:38:03.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:03 smithi177 ceph-mon[36034]: pgmap v390: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T00:38:03.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:03 smithi204 ceph-mon[36345]: pgmap v390: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T00:38:03.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:03 smithi123 ceph-mon[31487]: pgmap v390: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T00:38:06.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:06 smithi177 ceph-mon[36034]: pgmap v391: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:06 smithi204 ceph-mon[36345]: pgmap v391: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:06 smithi123 ceph-mon[31487]: pgmap v391: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:08.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:08 smithi177 ceph-mon[36034]: pgmap v392: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:08 smithi204 ceph-mon[36345]: pgmap v392: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:08.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:08 smithi123 ceph-mon[31487]: pgmap v392: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:10 smithi177 ceph-mon[36034]: pgmap v393: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:10 smithi204 ceph-mon[36345]: pgmap v393: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:10 smithi123 ceph-mon[31487]: pgmap v393: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:12.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:12 smithi177 ceph-mon[36034]: pgmap v394: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:12 smithi204 ceph-mon[36345]: pgmap v394: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:12 smithi123 ceph-mon[31487]: pgmap v394: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 -- 172.21.15.123:0/980316981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1016d0 msgr2=0x7f252c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 --2- 172.21.15.123:0/980316981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1016d0 0x7f252c101af0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f251c009860 tx=0x7f251c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 -- 172.21.15.123:0/980316981 shutdown_connections 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 --2- 172.21.15.123:0/980316981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c103280 0x7f252c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 --2- 172.21.15.123:0/980316981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c1028c0 0x7f252c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 --2- 172.21.15.123:0/980316981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1016d0 0x7f252c101af0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 -- 172.21.15.123:0/980316981 >> 172.21.15.123:0/980316981 conn(0x7f252c0fcc60 msgr2=0x7f252c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:12.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 -- 172.21.15.123:0/980316981 shutdown_connections 2024-03-24T00:38:12.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.972+0000 7f2531200640 1 -- 172.21.15.123:0/980316981 wait complete. 2024-03-24T00:38:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 Processor -- start 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 -- start start 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1028c0 0x7f252c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 0x7f252c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c10d350 con 0x7f252c103280 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c10d1d0 con 0x7f252c1016d0 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.973+0000 7f2531200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c10d4d0 con 0x7f252c1028c0 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53630/0 (socket says 172.21.15.123:53630) 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 -- 172.21.15.123:0/3629405223 learned_addr learned my addr 172.21.15.123:0/3629405223 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529400640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1028c0 0x7f252c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f252a800640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 0x7f252c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 -- 172.21.15.123:0/3629405223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1028c0 msgr2=0x7f252c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1028c0 0x7f252c073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 -- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 msgr2=0x7f252c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 0x7f252c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 -- 172.21.15.123:0/3629405223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c074860 con 0x7f252c1016d0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f252a800640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 0x7f252c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.974+0000 7f2529e00640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f251c0349e0 tx=0x7f251c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.975+0000 7f2522a00640 1 -- 172.21.15.123:0/3629405223 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c041020 con 0x7f252c1016d0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.975+0000 7f2522a00640 1 -- 172.21.15.123:0/3629405223 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f251c02c9f0 con 0x7f252c1016d0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.975+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c1a19c0 con 0x7f252c1016d0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.975+0000 7f2522a00640 1 -- 172.21.15.123:0/3629405223 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c039040 con 0x7f252c1016d0 2024-03-24T00:38:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.975+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c1a1f60 con 0x7f252c1016d0 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2522a00640 1 -- 172.21.15.123:0/3629405223 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f251c044020 con 0x7f252c1016d0 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 Processor -- start 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 -- start start 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb785d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb78651d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb78203b0 con 0x564bb7862d40 2024-03-24T00:38:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb77c2830 con 0x564bb785dbd0 2024-03-24T00:38:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.978+0000 7f2533fb7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb776b5d0 con 0x564bb785d1e0 2024-03-24T00:38:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53636/0 (socket says 172.21.15.123:53636) 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 -- 172.21.15.123:0/4272010979 learned_addr learned my addr 172.21.15.123:0/4272010979 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f252a800640 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb785d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529400640 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb78651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 -- 172.21.15.123:0/4272010979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 msgr2=0x564bb785d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb785d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 -- 172.21.15.123:0/4272010979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 msgr2=0x564bb78651d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb78651d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 -- 172.21.15.123:0/4272010979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bb77832c0 con 0x564bb785dbd0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.979+0000 7f2529e00640 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f251c0f8320 tx=0x7f251c0f8410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8e3c4975f766a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2517400640 1 -- 172.21.15.123:0/4272010979 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0f22d0 con 0x564bb785dbd0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2517400640 1 -- 172.21.15.123:0/4272010979 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f251c0fd030 con 0x564bb785dbd0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2517400640 1 -- 172.21.15.123:0/4272010979 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c101700 con 0x564bb785dbd0 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2522a00640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 0x7f25100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2533fb7740 1 -- 172.21.15.123:0/4272010979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 msgr2=0x564bb78626d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2533fb7740 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f251c0f8320 tx=0x7f251c0f8410 comp rx=0 tx=0).stop 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.980+0000 7f2522a00640 1 -- 172.21.15.123:0/3629405223 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f251c0f0310 con 0x7f252c1016d0 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f252a800640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 0x7f25100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 -- 172.21.15.123:0/4272010979 shutdown_connections 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb78651d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb78626d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 --2- 172.21.15.123:0/4272010979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb785d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 -- 172.21.15.123:0/4272010979 >> 172.21.15.123:0/4272010979 conn(0x564bb7858750 msgr2=0x564bb7858b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 -- 172.21.15.123:0/4272010979 shutdown_connections 2024-03-24T00:38:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 -- 172.21.15.123:0/4272010979 wait complete. 2024-03-24T00:38:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f252a800640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 0x7f25100a4de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x564bb785e430 tx=0x7f2524009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.981+0000 7f2533fb7740 1 -- start start 2024-03-24T00:38:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 Processor -- start 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb777ea20 con 0x564bb785d1e0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb77be8d0 con 0x564bb785dbd0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bb7865ab0 con 0x564bb7862d40 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.982+0000 7f2533fb7740 10 client.0 _ll_register_callbacks cb 0x564bb77bb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53648/0 (socket says 172.21.15.123:53648) 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 -- 172.21.15.123:0/1876760046 learned_addr learned my addr 172.21.15.123:0/1876760046 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f252a800640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 -- 172.21.15.123:0/1876760046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 msgr2=0x564bb786e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 -- 172.21.15.123:0/1876760046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 msgr2=0x564bb78670f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 -- 172.21.15.123:0/1876760046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bb74d7210 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529e00640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f2529400640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2518012700 tx=0x7f2518012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fd7ca53410142 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.983+0000 7f250be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2518004020 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f25180041c0 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f251c0fd030 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25180043c0 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 10 client.25268 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564bb75a3620 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f251c0fd030 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2518003bf0 con 0x564bb785dbd0 2024-03-24T00:38:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 10 client.25268 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 10 client.25268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 20 client.25268 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb7872ea0 0x564bb78752f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 4 client.25268 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.984+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bb77cfd70 con 0x564bb7872ea0 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.985+0000 7f2529e00640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb7872ea0 0x564bb78752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.985+0000 7f2529e00640 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb7872ea0 0x564bb78752f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e602a707d840dcbc server_cookie=478e1a096667b33e in_seq=0 out_seq=0 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.985+0000 7f250be00640 10 client.25268 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f250be00640 1 -- 172.21.15.123:0/1876760046 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564bb77cfd70 con 0x564bb7872ea0 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f250be00640 10 client.25268 handle_command_reply: tid=0 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 client.25268 shutdown 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 msgr2=0x564bb7867630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2518012700 tx=0x7f2518012bc0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 shutdown_connections 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb7872ea0 0x564bb78752f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f250be00640 0 client.25268 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1876760046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 >> 172.21.15.123:0/1876760046 conn(0x564bb7858750 msgr2=0x564bb7857950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 shutdown_connections 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.986+0000 7f2533fb7740 1 -- 172.21.15.123:0/1876760046 wait complete. 2024-03-24T00:38:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:12.987+0000 7f2533fb7740 20 client.25268 trim_cache size 0 max 16384 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.012+0000 7f2533fb7740 1 Processor -- start 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 -- start start 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb786e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a8c90 con 0x564bb7862d40 2024-03-24T00:38:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1ab250 con 0x564bb785dbd0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2533fb7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a9290 con 0x564bb785d1e0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2529400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f252a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb786e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2529400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53654/0 (socket says 172.21.15.123:53654) 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2529400640 1 -- 172.21.15.123:0/1421893746 learned_addr learned my addr 172.21.15.123:0/1421893746 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.013+0000 7f2529e00640 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 -- 172.21.15.123:0/1421893746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 msgr2=0x564bb78670f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 -- 172.21.15.123:0/1421893746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 msgr2=0x564bb786e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb786e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 -- 172.21.15.123:0/1421893746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bb78680e0 con 0x564bb785dbd0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529e00640 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2529400640 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2518001eb0 tx=0x7f2518005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0248ea87f9427aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2516a00640 1 -- 172.21.15.123:0/1421893746 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2518003fe0 con 0x564bb785dbd0 2024-03-24T00:38:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2516a00640 1 -- 172.21.15.123:0/1421893746 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2518004180 con 0x564bb785dbd0 2024-03-24T00:38:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.014+0000 7f2516a00640 1 -- 172.21.15.123:0/1421893746 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25180035f0 con 0x564bb785dbd0 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 -- 172.21.15.123:0/1421893746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 msgr2=0x564bb7867630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2518001eb0 tx=0x7f2518005850 comp rx=0 tx=0).stop 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 -- 172.21.15.123:0/1421893746 shutdown_connections 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb7862d40 0x564bb786e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785dbd0 0x564bb7867630 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 --2- 172.21.15.123:0/1421893746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb785d1e0 0x564bb78670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 -- 172.21.15.123:0/1421893746 >> 172.21.15.123:0/1421893746 conn(0x564bb7858750 msgr2=0x564bb7858b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 -- 172.21.15.123:0/1421893746 shutdown_connections 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.015+0000 7f2533fb7740 1 -- 172.21.15.123:0/1421893746 wait complete. 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 -- start start 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 Processor -- start 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785dbd0 0x564bb786c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a98d0 con 0x564bb785dbd0 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1aabf0 con 0x564bb785d1e0 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a9f30 con 0x564bb7862d40 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2533fb7740 10 client.0 _ll_register_callbacks cb 0x7f2518009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.016+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785dbd0 0x564bb786c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53656/0 (socket says 172.21.15.123:53656) 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 -- 172.21.15.123:0/3306697064 learned_addr learned my addr 172.21.15.123:0/3306697064 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f252a800640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 -- 172.21.15.123:0/3306697064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 msgr2=0x564bb786d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 -- 172.21.15.123:0/3306697064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785dbd0 msgr2=0x564bb786c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785dbd0 0x564bb786c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 -- 172.21.15.123:0/3306697064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bb786da10 con 0x564bb785d1e0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2529e00640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f251c02cbc0 tx=0x7f251c02c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7478ed018c787e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f250be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0f3e60 con 0x564bb785d1e0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2518004180 con 0x564bb785d1e0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.017+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f251c0bad00 con 0x564bb785d1e0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f251c0b8f00 con 0x564bb785d1e0 2024-03-24T00:38:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 10 client.25274 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564bb78ce8e0 con 0x564bb785d1e0 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2518004180 con 0x564bb785d1e0 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f251c03c030 con 0x564bb785d1e0 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 10 client.25274 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 10 client.25274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 20 client.25274 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb78cefd0 0x564bb78d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 4 client.25274 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564bb77ca0c0 con 0x564bb78cefd0 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.018+0000 7f2529400640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb78cefd0 0x564bb78d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.019+0000 7f250be00640 10 client.25274 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.019+0000 7f2529400640 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb78cefd0 0x564bb78d13f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d729ee62271ebd2b server_cookie=453d8f0f8401969a in_seq=0 out_seq=0 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f250be00640 1 -- 172.21.15.123:0/3306697064 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564bb77ca0c0 con 0x564bb78cefd0 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f250be00640 10 client.25274 handle_command_reply: tid=0 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 client.25274 shutdown 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 msgr2=0x564bb786bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f251c02cbc0 tx=0x7f251c02c1f0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 shutdown_connections 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564bb78cefd0 0x564bb78d13f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564bb7862d40 0x564bb786d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bb785dbd0 0x564bb786c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f250be00640 0 client.25274 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 --2- 172.21.15.123:0/3306697064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bb785d1e0 0x564bb786bed0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 >> 172.21.15.123:0/3306697064 conn(0x564bb7858750 msgr2=0x564bb785ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 shutdown_connections 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 1 -- 172.21.15.123:0/3306697064 wait complete. 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.030+0000 7f2533fb7740 20 client.25274 trim_cache size 0 max 16384 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:38:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:38:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:38:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 msgr2=0x7f25100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 0x7f25100a4de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x564bb785e430 tx=0x7f2524009500 comp rx=0 tx=0).stop 2024-03-24T00:38:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 msgr2=0x7f252c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f251c0349e0 tx=0x7f251c032430 comp rx=0 tx=0).stop 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 shutdown_connections 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f25100a2920 0x7f25100a4de0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f252c103280 0x7f252c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f252c1028c0 0x7f252c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 --2- 172.21.15.123:0/3629405223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f252c1016d0 0x7f252c075450 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 >> 172.21.15.123:0/3629405223 conn(0x7f252c0fcc60 msgr2=0x7f252c105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 shutdown_connections 2024-03-24T00:38:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:13.031+0000 7f2531200640 1 -- 172.21.15.123:0/3629405223 wait complete. 2024-03-24T00:38:14.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:14 smithi123 ceph-mon[31487]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:38:14.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:14 smithi177 ceph-mon[36034]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:38:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:14 smithi204 ceph-mon[36345]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T00:38:16.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:16 smithi177 ceph-mon[36034]: pgmap v396: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-24T00:38:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:16 smithi204 ceph-mon[36345]: pgmap v396: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-24T00:38:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:16 smithi123 ceph-mon[31487]: pgmap v396: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-24T00:38:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:18 smithi204 ceph-mon[36345]: pgmap v397: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:18 smithi123 ceph-mon[31487]: pgmap v397: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:18 smithi177 ceph-mon[36034]: pgmap v397: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:20 smithi204 ceph-mon[36345]: pgmap v398: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:20 smithi123 ceph-mon[31487]: pgmap v398: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:20 smithi177 ceph-mon[36034]: pgmap v398: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:22 smithi204 ceph-mon[36345]: pgmap v399: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:22 smithi123 ceph-mon[31487]: pgmap v399: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:22 smithi177 ceph-mon[36034]: pgmap v399: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:24.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:24 smithi123 ceph-mon[31487]: pgmap v400: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:24 smithi204 ceph-mon[36345]: pgmap v400: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:24 smithi177 ceph-mon[36034]: pgmap v400: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 -- 172.21.15.123:0/3370235362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 msgr2=0x7f869c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3370235362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c108120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f868c009900 tx=0x7f868c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 -- 172.21.15.123:0/3370235362 shutdown_connections 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3370235362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c1098b0 0x7f869c110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3370235362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c108ef0 0x7f869c109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3370235362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c108120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 -- 172.21.15.123:0/3370235362 >> 172.21.15.123:0/3370235362 conn(0x7f869c076cb0 msgr2=0x7f869c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 -- 172.21.15.123:0/3370235362 shutdown_connections 2024-03-24T00:38:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.169+0000 7f86a2a00640 1 -- 172.21.15.123:0/3370235362 wait complete. 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.170+0000 7f86a2a00640 1 Processor -- start 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.170+0000 7f86a2a00640 1 -- start start 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 0x7f869c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 0x7f869c1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1139d0 con 0x7f869c1098b0 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c113850 con 0x7f869c107d00 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a2a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c113b50 con 0x7f869c108ef0 2024-03-24T00:38:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37208/0 (socket says 172.21.15.123:37208) 2024-03-24T00:38:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f86a0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 0x7f869c1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f869be00640 1 -- 172.21.15.123:0/3242357954 learned_addr learned my addr 172.21.15.123:0/3242357954 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.171+0000 7f8693e00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 0x7f869c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 -- 172.21.15.123:0/3242357954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 msgr2=0x7f869c19e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 0x7f869c19e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 -- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 msgr2=0x7f869c1a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 0x7f869c1a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 -- 172.21.15.123:0/3242357954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c1a5c30 con 0x7f869c107d00 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f8693e00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 0x7f869c19e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 0x7f869c1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f869be00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f868c000c00 tx=0x7f868c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f8699600640 1 -- 172.21.15.123:0/3242357954 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c043020 con 0x7f869c107d00 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f8699600640 1 -- 172.21.15.123:0/3242357954 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c0054e0 con 0x7f869c107d00 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f8699600640 1 -- 172.21.15.123:0/3242357954 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c003c40 con 0x7f869c107d00 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c1a5ec0 con 0x7f869c107d00 2024-03-24T00:38:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.172+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c1a62d0 con 0x7f869c107d00 2024-03-24T00:38:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.175+0000 7f8699600640 1 -- 172.21.15.123:0/3242357954 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f868c046020 con 0x7f869c107d00 2024-03-24T00:38:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 Processor -- start 2024-03-24T00:38:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 -- start start 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a5575536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a5575561d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5575113b0 con 0x55a55754e1e0 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5574b3830 con 0x55a55754ebd0 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a579a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55745c5d0 con 0x55a557553d40 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a5575536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34824/0 (socket says 172.21.15.123:34824) 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.176+0000 7f86a0a00640 1 -- 172.21.15.123:0/3896946866 learned_addr learned my addr 172.21.15.123:0/3896946866 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f8693e00640 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a5575561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f8699600640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 0x7f86800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 -- 172.21.15.123:0/3896946866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 msgr2=0x55a5575561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a5575561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 -- 172.21.15.123:0/3896946866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 msgr2=0x55a5575536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a5575536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 -- 172.21.15.123:0/3896946866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5574742c0 con 0x55a55754e1e0 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f8699600640 1 -- 172.21.15.123:0/3242357954 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f868c0f0ac0 con 0x7f869c107d00 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 0x7f86800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.177+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8694011790 tx=0x7f8694011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6b993ba68512af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.178+0000 7f8691200640 1 -- 172.21.15.123:0/3896946866 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694020070 con 0x55a55754e1e0 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.178+0000 7f8691200640 1 -- 172.21.15.123:0/3896946866 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f869400fc20 con 0x55a55754e1e0 2024-03-24T00:38:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.178+0000 7f8691200640 1 -- 172.21.15.123:0/3896946866 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869401b9a0 con 0x55a55754e1e0 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- 172.21.15.123:0/3896946866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 msgr2=0x55a55754e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 0x7f86800a4de0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f869401d000 tx=0x7f869400e780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8694011790 tx=0x7f8694011c50 comp rx=0 tx=0).stop 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- 172.21.15.123:0/3896946866 shutdown_connections 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a5575561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a5575536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 --2- 172.21.15.123:0/3896946866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754e600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- 172.21.15.123:0/3896946866 >> 172.21.15.123:0/3896946866 conn(0x55a557549750 msgr2=0x55a557549b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- 172.21.15.123:0/3896946866 shutdown_connections 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- 172.21.15.123:0/3896946866 wait complete. 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 -- start start 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.179+0000 7f86a579a740 1 Processor -- start 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a557558820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a55755f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55746fa20 con 0x55a55754e1e0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5574af8d0 con 0x55a55754ebd0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a557556cf0 con 0x55a557553d40 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a579a740 10 client.0 _ll_register_callbacks cb 0x55a5574ac1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f869be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f8693e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a557558820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f869be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34832/0 (socket says 172.21.15.123:34832) 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f869be00640 1 -- 172.21.15.123:0/2086533172 learned_addr learned my addr 172.21.15.123:0/2086533172 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.180+0000 7f86a0a00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a55755f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 -- 172.21.15.123:0/2086533172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 msgr2=0x55a55755f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a55755f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 -- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 msgr2=0x55a557558820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a557558820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 -- 172.21.15.123:0/2086533172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5571c8210 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f86a0a00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a55755f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f8693e00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a557558820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f869be00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b9870 tx=0x7f868c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5d8d8dd07dfa3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f8689600640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0bcd70 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f868c0ee680 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f869400fc20 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f868c004250 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f86a579a740 10 client.15183 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.181+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a557294620 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f869400fc20 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f868c0c1160 con 0x55a55754e1e0 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 10 client.15183 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 10 client.15183 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 20 client.15183 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575640e0 0x55a557566510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 4 client.15183 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5574c0d70 con 0x55a5575640e0 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.182+0000 7f8693e00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575640e0 0x55a557566510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.183+0000 7f8693e00640 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575640e0 0x55a557566510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b01e6cd4fa7f366a server_cookie=97220ea11d57041b in_seq=0 out_seq=0 2024-03-24T00:38:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.183+0000 7f8689600640 10 client.15183 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.183+0000 7f8689600640 1 -- 172.21.15.123:0/2086533172 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a5574c0d70 con 0x55a5575640e0 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.183+0000 7f8689600640 10 client.15183 handle_command_reply: tid=0 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.183+0000 7f86a579a740 1 client.15183 shutdown 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 msgr2=0x55a5575582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b9870 tx=0x7f868c009620 comp rx=0 tx=0).stop 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 shutdown_connections 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575640e0 0x55a557566510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a557553d40 0x55a55755f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754ebd0 0x55a557558820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 --2- 172.21.15.123:0/2086533172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a5575582e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f8689600640 0 client.15183 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 >> 172.21.15.123:0/2086533172 conn(0x55a557549750 msgr2=0x55a557548950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 shutdown_connections 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 1 -- 172.21.15.123:0/2086533172 wait complete. 2024-03-24T00:38:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.184+0000 7f86a579a740 20 client.15183 trim_cache size 0 max 16384 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 Processor -- start 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 -- start start 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1ad0b0 con 0x55a55754e1e0 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1af670 con 0x55a557553d40 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.199+0000 7f86a579a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1ad6b0 con 0x55a55754ebd0 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f869be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37238/0 (socket says 172.21.15.123:37238) 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f869be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34842/0 (socket says 172.21.15.123:34842) 2024-03-24T00:38:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 -- 172.21.15.123:0/3028912795 learned_addr learned my addr 172.21.15.123:0/3028912795 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f8693e00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 -- 172.21.15.123:0/3028912795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 msgr2=0x55a55755f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 -- 172.21.15.123:0/3028912795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 msgr2=0x55a55754f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 -- 172.21.15.123:0/3028912795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a557566b90 con 0x55a557553d40 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f8693e00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f869be00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.200+0000 7f86a0a00640 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f869400f790 tx=0x7f8694028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36bab839de3faa34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f868be00640 1 -- 172.21.15.123:0/3028912795 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694020040 con 0x55a557553d40 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f868be00640 1 -- 172.21.15.123:0/3028912795 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8694035070 con 0x55a557553d40 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f868be00640 1 -- 172.21.15.123:0/3028912795 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400fd90 con 0x55a557553d40 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 -- 172.21.15.123:0/3028912795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 msgr2=0x55a55754c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f869400f790 tx=0x7f8694028000 comp rx=0 tx=0).stop 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 -- 172.21.15.123:0/3028912795 shutdown_connections 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a557553d40 0x55a55754c9b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 --2- 172.21.15.123:0/3028912795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a55754e1e0 0x55a55754f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 -- 172.21.15.123:0/3028912795 >> 172.21.15.123:0/3028912795 conn(0x55a557549750 msgr2=0x55a557553430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 -- 172.21.15.123:0/3028912795 shutdown_connections 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.201+0000 7f86a579a740 1 -- 172.21.15.123:0/3028912795 wait complete. 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 -- start start 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 Processor -- start 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a557553d40 0x55a55755e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1adcf0 con 0x55a557553d40 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1af010 con 0x55a55754e1e0 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1ae350 con 0x55a55754ebd0 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.202+0000 7f86a579a740 10 client.0 _ll_register_callbacks cb 0x7f868c0f40f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37246/0 (socket says 172.21.15.123:37246) 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 -- 172.21.15.123:0/118058105 learned_addr learned my addr 172.21.15.123:0/118058105 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f86a0a00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a557553d40 0x55a55755e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f8693e00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 -- 172.21.15.123:0/118058105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 msgr2=0x55a55755d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 -- 172.21.15.123:0/118058105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a557553d40 msgr2=0x55a55755e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a557553d40 0x55a55755e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 -- 172.21.15.123:0/118058105 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a55755eae0 con 0x55a55754e1e0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f869be00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f868c0bd3a0 tx=0x7f868c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83dfddc02d31ac50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f8689600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0bcd70 con 0x55a55754e1e0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f868c0c5720 con 0x55a55754e1e0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8694035070 con 0x55a55754e1e0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f868c0c5c00 con 0x55a55754e1e0 2024-03-24T00:38:26.208 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f86a579a740 10 client.25292 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.203+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a5575bfb10 con 0x55a55754e1e0 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8694035070 con 0x55a55754e1e0 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f868c0dd0b0 con 0x55a55754e1e0 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 10 client.25292 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 10 client.25292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 20 client.25292 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575c0160 0x55a5575c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 4 client.25292 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a5574bb0c0 con 0x55a5575c0160 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.204+0000 7f8693e00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575c0160 0x55a5575c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.205+0000 7f8693e00640 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575c0160 0x55a5575c2580 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f1e9ae4481d6dc7 server_cookie=577ed930b3d03b83 in_seq=0 out_seq=0 2024-03-24T00:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.205+0000 7f8689600640 10 client.25292 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.210+0000 7f8689600640 1 -- 172.21.15.123:0/118058105 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a5574bb0c0 con 0x55a5575c0160 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.210+0000 7f8689600640 10 client.25292 handle_command_reply: tid=0 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.210+0000 7f86a579a740 1 client.25292 shutdown 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 msgr2=0x55a55755cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f868c0bd3a0 tx=0x7f868c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 shutdown_connections 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a557553d40 0x55a55755e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a5575c0160 0x55a5575c2580 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a55754ebd0 0x55a55755d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 --2- 172.21.15.123:0/118058105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a55754e1e0 0x55a55755cff0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 >> 172.21.15.123:0/118058105 conn(0x55a557549750 msgr2=0x55a557553000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f8689600640 0 client.25292 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 shutdown_connections 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 1 -- 172.21.15.123:0/118058105 wait complete. 2024-03-24T00:38:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.211+0000 7f86a579a740 20 client.25292 trim_cache size 0 max 16384 2024-03-24T00:38:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:38:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:38:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:38:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 msgr2=0x7f86800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 0x7f86800a4de0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f869401d000 tx=0x7f869400e780 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 msgr2=0x7f869c19e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f868c000c00 tx=0x7f868c02c650 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 shutdown_connections 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2920 0x7f86800a4de0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c1098b0 0x7f869c1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c108ef0 0x7f869c19e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 --2- 172.21.15.123:0/3242357954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107d00 0x7f869c19e010 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 >> 172.21.15.123:0/3242357954 conn(0x7f869c076cb0 msgr2=0x7f869c10c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 shutdown_connections 2024-03-24T00:38:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:26.212+0000 7f86a2a00640 1 -- 172.21.15.123:0/3242357954 wait complete. 2024-03-24T00:38:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:26 smithi204 ceph-mon[36345]: pgmap v401: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:26 smithi123 ceph-mon[31487]: pgmap v401: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:26 smithi177 ceph-mon[36034]: pgmap v401: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T00:38:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:28 smithi204 ceph-mon[36345]: pgmap v402: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:28 smithi123 ceph-mon[31487]: pgmap v402: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:28 smithi177 ceph-mon[36034]: pgmap v402: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:30.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:38:30.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 -- 172.21.15.123:0/1944796956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 msgr2=0x7f12dc102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 --2- 172.21.15.123:0/1944796956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc102bf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009860 tx=0x7f12c402bb70 comp rx=0 tx=0).stop 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 -- 172.21.15.123:0/1944796956 shutdown_connections 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 --2- 172.21.15.123:0/1944796956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 0x7f12dc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 --2- 172.21.15.123:0/1944796956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc102bf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 --2- 172.21.15.123:0/1944796956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 -- 172.21.15.123:0/1944796956 >> 172.21.15.123:0/1944796956 conn(0x7f12dc0fcb10 msgr2=0x7f12dc0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 -- 172.21.15.123:0/1944796956 shutdown_connections 2024-03-24T00:38:30.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.566+0000 7f12e0c00640 1 -- 172.21.15.123:0/1944796956 wait complete. 2024-03-24T00:38:30.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.567+0000 7f12e0c00640 1 Processor -- start 2024-03-24T00:38:30.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 -- start start 2024-03-24T00:38:30.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 0x7f12dc1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10d210 con 0x7f12dc103130 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10d090 con 0x7f12dc102770 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12e0c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10d390 con 0x7f12dc101580 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:55948/0 (socket says 172.21.15.123:55948) 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37266/0 (socket says 172.21.15.123:37266) 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12db400640 1 -- 172.21.15.123:0/2214247232 learned_addr learned my addr 172.21.15.123:0/2214247232 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.568+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 0x7f12dc1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 -- 172.21.15.123:0/2214247232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 msgr2=0x7f12dc199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 -- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 msgr2=0x7f12dc1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 0x7f12dc1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 -- 172.21.15.123:0/2214247232 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc1a16d0 con 0x7f12dc102770 2024-03-24T00:38:30.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12daa00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12c40057c0 tx=0x7f12c4034760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12d3e00640 1 -- 172.21.15.123:0/2214247232 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4041020 con 0x7f12dc102770 2024-03-24T00:38:30.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12d3e00640 1 -- 172.21.15.123:0/2214247232 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c402c9f0 con 0x7f12dc102770 2024-03-24T00:38:30.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12dc1a1960 con 0x7f12dc102770 2024-03-24T00:38:30.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.569+0000 7f12d3e00640 1 -- 172.21.15.123:0/2214247232 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4039040 con 0x7f12dc102770 2024-03-24T00:38:30.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.570+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12dc1a1e50 con 0x7f12dc102770 2024-03-24T00:38:30.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:30 smithi204 ceph-mon[36345]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 511 B/s wr, 3 op/s 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 Processor -- start 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 -- start start 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12d3e00640 1 -- 172.21.15.123:0/2214247232 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f12c4044020 con 0x7f12dc102770 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f905f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9064540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f8fc4250 con 0x55d8f905fa40 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f8f6ca30 con 0x55d8f9064bb0 2024-03-24T00:38:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.574+0000 7f12e4e38740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f8fc02f0 con 0x55d8f905f050 2024-03-24T00:38:30.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12db400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9064540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37272/0 (socket says 172.21.15.123:37272) 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12db400640 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9064540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34882/0 (socket says 172.21.15.123:34882) 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12dbe00640 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f905f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 -- 172.21.15.123:0/247100096 learned_addr learned my addr 172.21.15.123:0/247100096 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 -- 172.21.15.123:0/247100096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 msgr2=0x55d8f905f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f905f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 -- 172.21.15.123:0/247100096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 msgr2=0x55d8f9064540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9064540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 -- 172.21.15.123:0/247100096 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8f8f57690 con 0x55d8f9064bb0 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.575+0000 7f12daa00640 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f12c40f8210 tx=0x7f12c40f8300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a74abe059aded2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12d3e00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 0x7f12c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12cbe00640 1 -- 172.21.15.123:0/247100096 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40b8e10 con 0x55d8f9064bb0 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12cbe00640 1 -- 172.21.15.123:0/247100096 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c40fd030 con 0x55d8f9064bb0 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12cbe00640 1 -- 172.21.15.123:0/247100096 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40bbfc0 con 0x55d8f9064bb0 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 0x7f12c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12d3e00640 1 -- 172.21.15.123:0/2214247232 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f12c40f0220 con 0x7f12dc102770 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12e4e38740 1 -- 172.21.15.123:0/247100096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 msgr2=0x55d8f9067040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.576+0000 7f12e4e38740 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f12c40f8210 tx=0x7f12c40f8300 comp rx=0 tx=0).stop 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 0x7f12c00a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55d8f90602a0 tx=0x7f12d4010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 -- 172.21.15.123:0/247100096 shutdown_connections 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9067040 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9064540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 --2- 172.21.15.123:0/247100096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f905f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 -- 172.21.15.123:0/247100096 >> 172.21.15.123:0/247100096 conn(0x55d8f9027200 msgr2=0x55d8f905a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 -- 172.21.15.123:0/247100096 shutdown_connections 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 -- 172.21.15.123:0/247100096 wait complete. 2024-03-24T00:38:30.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 -- start start 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.577+0000 7f12e4e38740 1 Processor -- start 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9069220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f9070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f9067920 con 0x55d8f905fa40 2024-03-24T00:38:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f8f7fe80 con 0x55d8f905f050 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8f9067aa0 con 0x55d8f9064bb0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12e4e38740 10 client.0 _ll_register_callbacks cb 0x55d8f8f92130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37288/0 (socket says 172.21.15.123:37288) 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12daa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 -- 172.21.15.123:0/2402758706 learned_addr learned my addr 172.21.15.123:0/2402758706 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f9070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 -- 172.21.15.123:0/2402758706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 msgr2=0x55d8f9070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f9070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 -- 172.21.15.123:0/2402758706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 msgr2=0x55d8f9069220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9069220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.578+0000 7f12db400640 1 -- 172.21.15.123:0/2402758706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8f8da27d0 con 0x55d8f905f050 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12db400640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12cc012760 tx=0x7f12cc012c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbac097b6898519b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:30.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc004030 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12c40fd030 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f12cc0041d0 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12cc0043d0 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12e4e38740 10 client.25304 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d8f8cd83f0 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.579+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c40fd030 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f12cc003bf0 con 0x55d8f905f050 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 10 client.25304 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 10 client.25304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 20 client.25304 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f9074ac0 0x55d8f9076f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 4 client.25304 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8f8fd1790 con 0x55d8f9074ac0 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12daa00640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f9074ac0 0x55d8f9076f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12daa00640 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f9074ac0 0x55d8f9076f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e94d20e1e839bb8 server_cookie=ca06109c80444274 in_seq=0 out_seq=0 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.580+0000 7f12c8c00640 10 client.25304 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12c8c00640 1 -- 172.21.15.123:0/2402758706 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d8f8fd1790 con 0x55d8f9074ac0 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12c8c00640 10 client.25304 handle_command_reply: tid=0 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 client.25304 shutdown 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 msgr2=0x55d8f9068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12cc012760 tx=0x7f12cc012c20 comp rx=0 tx=0).stop 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 shutdown_connections 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f9074ac0 0x55d8f9076f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f9070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9069220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12c8c00640 0 client.25304 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2402758706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905f050 0x55d8f9068ce0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.581+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 >> 172.21.15.123:0/2402758706 conn(0x55d8f9027200 msgr2=0x55d8f9059bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.582+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 shutdown_connections 2024-03-24T00:38:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.582+0000 7f12e4e38740 1 -- 172.21.15.123:0/2402758706 wait complete. 2024-03-24T00:38:30.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.582+0000 7f12e4e38740 20 client.25304 trim_cache size 0 max 16384 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 Processor -- start 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 -- start start 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f9060c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9070220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1a8b40 con 0x55d8f905fa40 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1ab100 con 0x55d8f9064bb0 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.599+0000 7f12e4e38740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1a9140 con 0x55d8f905f050 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37292/0 (socket says 172.21.15.123:37292) 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 -- 172.21.15.123:0/3539048537 learned_addr learned my addr 172.21.15.123:0/3539048537 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12daa00640 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12db400640 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f9060c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 -- 172.21.15.123:0/3539048537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 msgr2=0x55d8f9060c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f9060c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 -- 172.21.15.123:0/3539048537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 msgr2=0x55d8f9070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9070220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 -- 172.21.15.123:0/3539048537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8f8c980f0 con 0x55d8f9064bb0 2024-03-24T00:38:30.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12dbe00640 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12d4004070 tx=0x7f12d4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35e6032b481ce7a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.600+0000 7f12cb400640 1 -- 172.21.15.123:0/3539048537 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d400db10 con 0x55d8f9064bb0 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12cb400640 1 -- 172.21.15.123:0/3539048537 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d402b070 con 0x55d8f9064bb0 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12cb400640 1 -- 172.21.15.123:0/3539048537 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4026680 con 0x55d8f9064bb0 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 -- 172.21.15.123:0/3539048537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 msgr2=0x55d8f9059a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12d4004070 tx=0x7f12d4016000 comp rx=0 tx=0).stop 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 -- 172.21.15.123:0/3539048537 shutdown_connections 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f9064bb0 0x55d8f9059a40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905fa40 0x55d8f9070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 --2- 172.21.15.123:0/3539048537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f905f050 0x55d8f9060c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 -- 172.21.15.123:0/3539048537 >> 172.21.15.123:0/3539048537 conn(0x55d8f9027200 msgr2=0x55d8f9059330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 -- 172.21.15.123:0/3539048537 shutdown_connections 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.601+0000 7f12e4e38740 1 -- 172.21.15.123:0/3539048537 wait complete. 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 -- start start 2024-03-24T00:38:30.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 Processor -- start 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905f050 0x55d8f906fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f906e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1a9780 con 0x55d8f905f050 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1aaaa0 con 0x55d8f905fa40 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc1a9de0 con 0x55d8f9064bb0 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12e4e38740 10 client.0 _ll_register_callbacks cb 0x7f12cc0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12db400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905f050 0x55d8f906fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.602+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37306/0 (socket says 172.21.15.123:37306) 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 -- 172.21.15.123:0/2188078387 learned_addr learned my addr 172.21.15.123:0/2188078387 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f906e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 -- 172.21.15.123:0/2188078387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 msgr2=0x55d8f906e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f906e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 -- 172.21.15.123:0/2188078387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905f050 msgr2=0x55d8f906fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905f050 0x55d8f906fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 -- 172.21.15.123:0/2188078387 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8f906ea90 con 0x55d8f905fa40 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12dbe00640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f906e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12daa00640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12c402cbc0 tx=0x7f12c402bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff4b9648f9c4a41b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40bc2f0 con 0x55d8f905fa40 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f12c4032e00 con 0x55d8f905fa40 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.603+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12d402b070 con 0x55d8f905fa40 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12c40bc4d0 con 0x55d8f905fa40 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 10 client.25310 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:30.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d8f90d9790 con 0x55d8f905fa40 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d402b070 con 0x55d8f905fa40 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f12c403c030 con 0x55d8f905fa40 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 10 client.25310 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 10 client.25310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 20 client.25310 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f90da020 0x55d8f90dc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 4 client.25310 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d8f8fcbae0 con 0x55d8f90da020 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.604+0000 7f12db400640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f90da020 0x55d8f90dc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12db400640 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f90da020 0x55d8f90dc410 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=121a63a73c9fd924 server_cookie=de698cd34d966444 in_seq=0 out_seq=0 2024-03-24T00:38:30.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12c8c00640 10 client.25310 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:30.609 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:30 smithi123 ceph-mon[31487]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 511 B/s wr, 3 op/s 2024-03-24T00:38:30.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12c8c00640 1 -- 172.21.15.123:0/2188078387 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d8f8fcbae0 con 0x55d8f90da020 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12c8c00640 10 client.25310 handle_command_reply: tid=0 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 client.25310 shutdown 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 msgr2=0x55d8f9070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12c402cbc0 tx=0x7f12c402bb70 comp rx=0 tx=0).stop 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 shutdown_connections 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8f90da020 0x55d8f90dc410 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8f9064bb0 0x55d8f906e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f905fa40 0x55d8f9070190 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12c8c00640 0 client.25310 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 --2- 172.21.15.123:0/2188078387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8f905f050 0x55d8f906fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 >> 172.21.15.123:0/2188078387 conn(0x55d8f9027200 msgr2=0x55d8f9060af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.605+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 shutdown_connections 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e4e38740 1 -- 172.21.15.123:0/2188078387 wait complete. 2024-03-24T00:38:30.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e4e38740 20 client.25310 trim_cache size 0 max 16384 2024-03-24T00:38:30.612 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:38:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:38:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:38:30.612 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 msgr2=0x7f12c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 0x7f12c00a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55d8f90602a0 tx=0x7f12d4010040 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 msgr2=0x7f12dc199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12c40057c0 tx=0x7f12c4034760 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 shutdown_connections 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f12c00a2800 0x7f12c00a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12dc103130 0x7f12dc1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12dc102770 0x7f12dc199ff0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 --2- 172.21.15.123:0/2214247232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12dc101580 0x7f12dc199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 >> 172.21.15.123:0/2214247232 conn(0x7f12dc0fcb10 msgr2=0x7f12dc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.606+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 shutdown_connections 2024-03-24T00:38:30.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.607+0000 7f12e0c00640 1 -- 172.21.15.123:0/2214247232 wait complete. 2024-03-24T00:38:30.624 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:38:30.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:30 smithi177 ceph-mon[36034]: pgmap v403: 97 pgs: 97 active+clean; 652 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 511 B/s wr, 3 op/s 2024-03-24T00:38:30.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 -- 172.21.15.123:0/963379892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c102160 msgr2=0x7fa87c104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 --2- 172.21.15.123:0/963379892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c102160 0x7fa87c104550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa870009900 tx=0x7fa87002bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 -- 172.21.15.123:0/963379892 shutdown_connections 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 --2- 172.21.15.123:0/963379892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c1073c0 0x7fa87c109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 --2- 172.21.15.123:0/963379892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c104a90 0x7fa87c106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 --2- 172.21.15.123:0/963379892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c102160 0x7fa87c104550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 -- 172.21.15.123:0/963379892 >> 172.21.15.123:0/963379892 conn(0x7fa87c0fbb50 msgr2=0x7fa87c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 -- 172.21.15.123:0/963379892 shutdown_connections 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.828+0000 7fa884800640 1 -- 172.21.15.123:0/963379892 wait complete. 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 Processor -- start 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 -- start start 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c102160 0x7fa87c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 0x7fa87c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c10d3b0 con 0x7fa87c102160 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c10d230 con 0x7fa87c104a90 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.829+0000 7fa884800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c10d530 con 0x7fa87c1073c0 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c102160 0x7fa87c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37328/0 (socket says 172.21.15.123:37328) 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 -- 172.21.15.123:0/2388864695 learned_addr learned my addr 172.21.15.123:0/2388864695 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa882800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 0x7fa87c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 -- 172.21.15.123:0/2388864695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 msgr2=0x7fa87c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 0x7fa87c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 -- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c102160 msgr2=0x7fa87c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c102160 0x7fa87c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 -- 172.21.15.123:0/2388864695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c19f640 con 0x7fa87c104a90 2024-03-24T00:38:30.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa882800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 0x7fa87c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa881400640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e9c0 tx=0x7fa86c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00cd80 con 0x7fa87c104a90 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa86c005500 con 0x7fa87c104a90 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c19f930 con 0x7fa87c104a90 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.830+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c010690 con 0x7fa87c104a90 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.831+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c19fed0 con 0x7fa87c104a90 2024-03-24T00:38:30.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.832+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c0fd640 con 0x7fa87c104a90 2024-03-24T00:38:30.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.834+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa86c0046c0 con 0x7fa87c104a90 2024-03-24T00:38:30.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.835+0000 7fa876a00640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 0x7fa8640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.835+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa86c0cee40 con 0x7fa87c104a90 2024-03-24T00:38:30.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.836+0000 7fa881e00640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 0x7fa8640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.836+0000 7fa881e00640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 0x7fa8640a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa87002c240 tx=0x7fa87002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:30.839+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa86c09be90 con 0x7fa87c104a90 2024-03-24T00:38:31.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.080+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa87c0622a0 con 0x7fa87c104a90 2024-03-24T00:38:31.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.081+0000 7fa876a00640 1 -- 172.21.15.123:0/2388864695 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fa86c09b790 con 0x7fa87c104a90 2024-03-24T00:38:31.082 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:38:31.082 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:38:31.082 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 msgr2=0x7fa8640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 0x7fa8640a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa87002c240 tx=0x7fa87002df90 comp rx=0 tx=0).stop 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 msgr2=0x7fa87c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e9c0 tx=0x7fa86c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 shutdown_connections 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa87c1073c0 0x7fa87c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa87c104a90 0x7fa87c197f60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8640a2880 0x7fa8640a4d40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 --2- 172.21.15.123:0/2388864695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa87c102160 0x7fa87c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 >> 172.21.15.123:0/2388864695 conn(0x7fa87c0fbb50 msgr2=0x7fa87c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 shutdown_connections 2024-03-24T00:38:31.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.084+0000 7fa884800640 1 -- 172.21.15.123:0/2388864695 wait complete. 2024-03-24T00:38:31.104 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:38:31.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 -- 172.21.15.123:0/2391337302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5480105370 msgr2=0x7f548010bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 --2- 172.21.15.123:0/2391337302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5480105370 0x7f548010bc00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5478009ca0 tx=0x7f547802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 -- 172.21.15.123:0/2391337302 shutdown_connections 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 --2- 172.21.15.123:0/2391337302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5480105370 0x7f548010bc00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 --2- 172.21.15.123:0/2391337302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54801049b0 0x7f5480104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 --2- 172.21.15.123:0/2391337302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 -- 172.21.15.123:0/2391337302 >> 172.21.15.123:0/2391337302 conn(0x7f54800fed50 msgr2=0x7f5480101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 -- 172.21.15.123:0/2391337302 shutdown_connections 2024-03-24T00:38:31.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.322+0000 7f5484a00640 1 -- 172.21.15.123:0/2391337302 wait complete. 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 Processor -- start 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 -- start start 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 0x7f5480199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5480105370 0x7f54801a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548010f440 con 0x7f54801037c0 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548010f2c0 con 0x7f54801049b0 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f5484a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548010f5c0 con 0x7f5480105370 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.323+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 0x7f5480199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34944/0 (socket says 172.21.15.123:34944) 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547f400640 1 -- 172.21.15.123:0/2765734559 learned_addr learned my addr 172.21.15.123:0/2765734559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:31.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547fe00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5480105370 0x7f54801a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 -- 172.21.15.123:0/2765734559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5480105370 msgr2=0x7f54801a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5480105370 0x7f54801a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 -- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 msgr2=0x7f5480199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 -- 172.21.15.123:0/2765734559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54801a1670 con 0x7f54801049b0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547f400640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f547ea00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 0x7f5480199f90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f546800e9f0 tx=0x7f546800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f5477e00640 1 -- 172.21.15.123:0/2765734559 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546800ccc0 con 0x7f54801049b0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f5477e00640 1 -- 172.21.15.123:0/2765734559 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5468005020 con 0x7f54801049b0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.324+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801a1960 con 0x7f54801049b0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.325+0000 7f5477e00640 1 -- 172.21.15.123:0/2765734559 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54680049e0 con 0x7f54801049b0 2024-03-24T00:38:31.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.325+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54801a1eb0 con 0x7f54801049b0 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.327+0000 7f5477e00640 1 -- 172.21.15.123:0/2765734559 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5468004c80 con 0x7f54801049b0 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.328+0000 7f5488d9a740 1 Processor -- start 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.328+0000 7f5488d9a740 1 -- start start 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.328+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.328+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698bfbc0 0x55fd698c0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5488d9a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd698232c0 con 0x55fd698bfbc0 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5488d9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd697cbc90 con 0x55fd698be9f0 2024-03-24T00:38:31.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5488d9a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd6981f360 con 0x55fd698c0580 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37352/0 (socket says 172.21.15.123:37352) 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 -- 172.21.15.123:0/841700821 learned_addr learned my addr 172.21.15.123:0/841700821 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547f400640 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698bfbc0 0x55fd698c0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547ea00640 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5477e00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 0x7f54640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 -- 172.21.15.123:0/841700821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 msgr2=0x55fd698c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 -- 172.21.15.123:0/841700821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698bfbc0 msgr2=0x55fd698c0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698bfbc0 0x55fd698c0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f547fe00640 1 -- 172.21.15.123:0/841700821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd697f0080 con 0x55fd698be9f0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.329+0000 7f5477e00640 1 -- 172.21.15.123:0/2765734559 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f54680cea90 con 0x7f54801049b0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.330+0000 7f547fe00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 0x7f54640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.330+0000 7f547fe00640 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5478005e30 tx=0x7f54780323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6902de085273fa45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.330+0000 7f546fe00640 1 -- 172.21.15.123:0/841700821 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478045030 con 0x55fd698be9f0 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.330+0000 7f546fe00640 1 -- 172.21.15.123:0/841700821 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5478032c90 con 0x55fd698be9f0 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.330+0000 7f546fe00640 1 -- 172.21.15.123:0/841700821 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478039040 con 0x55fd698be9f0 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 -- 172.21.15.123:0/841700821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 msgr2=0x55fd698bee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f547fe00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 0x7f54640a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5478045000 tx=0x7f5478004440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5478005e30 tx=0x7f54780323d0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 -- 172.21.15.123:0/841700821 shutdown_connections 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698bfbc0 0x55fd698c0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 --2- 172.21.15.123:0/841700821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698be9f0 0x55fd698bee10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 -- 172.21.15.123:0/841700821 >> 172.21.15.123:0/841700821 conn(0x55fd698b9de0 msgr2=0x55fd698ba1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 -- 172.21.15.123:0/841700821 shutdown_connections 2024-03-24T00:38:31.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.331+0000 7f5488d9a740 1 -- 172.21.15.123:0/841700821 wait complete. 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 -- start start 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 Processor -- start 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 0x55fd698c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd698c7590 con 0x55fd698be9f0 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd697df0d0 con 0x55fd698c0580 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd698c7710 con 0x55fd698bfbc0 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f5488d9a740 10 client.0 _ll_register_callbacks cb 0x55fd697e3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.332+0000 7f547fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37360/0 (socket says 172.21.15.123:37360) 2024-03-24T00:38:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34970/0 (socket says 172.21.15.123:34970) 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 -- 172.21.15.123:0/522012676 learned_addr learned my addr 172.21.15.123:0/522012676 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547ea00640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 0x55fd698c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 -- 172.21.15.123:0/522012676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 msgr2=0x55fd698c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 0x55fd698c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 -- 172.21.15.123:0/522012676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 msgr2=0x55fd698c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 -- 172.21.15.123:0/522012676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd695373f0 con 0x55fd698c0580 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547ea00640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 0x55fd698c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547f400640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f547fe00640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5478037040 tx=0x7f5478040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac8fa96065361597 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f546cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.333+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547805a020 con 0x55fd698c0580 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5478032c90 con 0x55fd698c0580 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5478003970 con 0x55fd698c0580 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5478039030 con 0x55fd698c0580 2024-03-24T00:38:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 10 client.25328 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55fd696017d0 con 0x55fd698c0580 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5478032c90 con 0x55fd698c0580 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f547803fb00 con 0x55fd698c0580 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 10 client.25328 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 10 client.25328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 20 client.25328 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd698d4290 0x55fd698d6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 4 client.25328 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.334+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd69830800 con 0x55fd698d4290 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.335+0000 7f547f400640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd698d4290 0x55fd698d6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.335+0000 7f547f400640 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd698d4290 0x55fd698d6680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=713dd56ee0193b7d server_cookie=f8ebaa5f0c51aded in_seq=0 out_seq=0 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.335+0000 7f546cc00640 10 client.25328 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f546cc00640 1 -- 172.21.15.123:0/522012676 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fd69830800 con 0x55fd698d4290 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f546cc00640 10 client.25328 handle_command_reply: tid=0 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 client.25328 shutdown 2024-03-24T00:38:31.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 msgr2=0x55fd698d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5478037040 tx=0x7f5478040000 comp rx=0 tx=0).stop 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 shutdown_connections 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698c0580 0x55fd698d0130 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd698d4290 0x55fd698d6680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698bfbc0 0x55fd698c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 --2- 172.21.15.123:0/522012676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f546cc00640 0 client.25328 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 >> 172.21.15.123:0/522012676 conn(0x55fd698b9de0 msgr2=0x55fd698b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 shutdown_connections 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 1 -- 172.21.15.123:0/522012676 wait complete. 2024-03-24T00:38:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.336+0000 7f5488d9a740 20 client.25328 trim_cache size 0 max 16384 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 Processor -- start 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 -- start start 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801a8b00 con 0x55fd698be9f0 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801ab0c0 con 0x55fd698bfbc0 2024-03-24T00:38:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f5488d9a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801a9100 con 0x55fd698c0580 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.350+0000 7f547ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34986/0 (socket says 172.21.15.123:34986) 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547f400640 1 -- 172.21.15.123:0/3062771588 learned_addr learned my addr 172.21.15.123:0/3062771588 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547fe00640 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 -- 172.21.15.123:0/3062771588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 msgr2=0x55fd698c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 -- 172.21.15.123:0/3062771588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 msgr2=0x55fd698c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 -- 172.21.15.123:0/3062771588 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd698bf640 con 0x55fd698bfbc0 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547f400640 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f547ea00640 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698d0050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f546800c6a0 tx=0x7f5468097750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0a9b2cfe7aa876 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f546f400640 1 -- 172.21.15.123:0/3062771588 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546801c040 con 0x55fd698bfbc0 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f546f400640 1 -- 172.21.15.123:0/3062771588 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546809bae0 con 0x55fd698bfbc0 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.351+0000 7f546f400640 1 -- 172.21.15.123:0/3062771588 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468005560 con 0x55fd698bfbc0 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- 172.21.15.123:0/3062771588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 msgr2=0x55fd698d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698d0050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f546800c6a0 tx=0x7f5468097750 comp rx=0 tx=0).stop 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- 172.21.15.123:0/3062771588 shutdown_connections 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698d0050 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 --2- 172.21.15.123:0/3062771588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- 172.21.15.123:0/3062771588 >> 172.21.15.123:0/3062771588 conn(0x55fd698b9de0 msgr2=0x55fd698bd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- 172.21.15.123:0/3062771588 shutdown_connections 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- 172.21.15.123:0/3062771588 wait complete. 2024-03-24T00:38:31.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.352+0000 7f5488d9a740 1 -- start start 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 Processor -- start 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801a9740 con 0x55fd698be9f0 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801aaa60 con 0x55fd698bfbc0 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801a9da0 con 0x55fd698c0580 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f5488d9a740 10 client.0 _ll_register_callbacks cb 0x7f5470009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34992/0 (socket says 172.21.15.123:34992) 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547f400640 1 -- 172.21.15.123:0/57770820 learned_addr learned my addr 172.21.15.123:0/57770820 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37376/0 (socket says 172.21.15.123:37376) 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.353+0000 7f547fe00640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 -- 172.21.15.123:0/57770820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 msgr2=0x55fd698cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 -- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 msgr2=0x55fd698ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 -- 172.21.15.123:0/57770820 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd698cf950 con 0x55fd698be9f0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547ea00640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f547f400640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54700058c0 tx=0x7f5470013f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8607e3cc47db91fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f546cc00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470026070 con 0x55fd698be9f0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5470004250 con 0x55fd698be9f0 2024-03-24T00:38:31.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f546809bae0 con 0x55fd698be9f0 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5470004450 con 0x55fd698be9f0 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f5488d9a740 10 client.15231 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.354+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55fd69930110 con 0x55fd698be9f0 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546809bae0 con 0x55fd698be9f0 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5470029020 con 0x55fd698be9f0 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 10 client.15231 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 10 client.15231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 20 client.15231 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd69930b20 0x55fd69932f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 4 client.15231 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:38:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fd6982ab50 con 0x55fd69930b20 2024-03-24T00:38:31.359 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:31 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2388864695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.355+0000 7f547ea00640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd69930b20 0x55fd69932f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f547ea00640 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd69930b20 0x55fd69932f40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dff6f75c1e1b5a9 server_cookie=502c660ae50e7904 in_seq=0 out_seq=0 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f546cc00640 10 client.15231 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f546cc00640 1 -- 172.21.15.123:0/57770820 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fd6982ab50 con 0x55fd69930b20 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f546cc00640 10 client.15231 handle_command_reply: tid=0 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 client.15231 shutdown 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 msgr2=0x55fd698cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54700058c0 tx=0x7f5470013f40 comp rx=0 tx=0).stop 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 shutdown_connections 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55fd69930b20 0x55fd69932f40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd698c0580 0x55fd698cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd698bfbc0 0x55fd698ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 --2- 172.21.15.123:0/57770820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd698be9f0 0x55fd698cde60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f546cc00640 0 client.15231 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:31.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.356+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 >> 172.21.15.123:0/57770820 conn(0x55fd698b9de0 msgr2=0x55fd698b88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 shutdown_connections 2024-03-24T00:38:31.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5488d9a740 1 -- 172.21.15.123:0/57770820 wait complete. 2024-03-24T00:38:31.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5488d9a740 20 client.15231 trim_cache size 0 max 16384 2024-03-24T00:38:31.362 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 msgr2=0x7f54640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 0x7f54640a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5478045000 tx=0x7f5478004440 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 msgr2=0x7f5480199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 0x7f5480199f90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f546800e9f0 tx=0x7f546800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 shutdown_connections 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5480105370 0x7f54801a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54801049b0 0x7f5480199f90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f54640a2880 0x7f54640a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 --2- 172.21.15.123:0/2765734559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54801037c0 0x7f5480199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.357+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 >> 172.21.15.123:0/2765734559 conn(0x7f54800fed50 msgr2=0x7f5480107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.358+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 shutdown_connections 2024-03-24T00:38:31.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:31.358+0000 7f5484a00640 1 -- 172.21.15.123:0/2765734559 wait complete. 2024-03-24T00:38:31.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:31 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2388864695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:38:31.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:31 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2388864695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:38:32.376 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:38:32.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:32.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:38:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:32 smithi204 ceph-mon[36345]: pgmap v404: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 -- 172.21.15.123:0/1082078534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b01038e0 msgr2=0x7f46b0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 --2- 172.21.15.123:0/1082078534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b01038e0 0x7f46b0103d00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009ca0 tx=0x7f46a002c100 comp rx=0 tx=0).stop 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 -- 172.21.15.123:0/1082078534 shutdown_connections 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 --2- 172.21.15.123:0/1082078534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b0105490 0x7f46b010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 --2- 172.21.15.123:0/1082078534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0104ad0 0x7f46b0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 --2- 172.21.15.123:0/1082078534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b01038e0 0x7f46b0103d00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 -- 172.21.15.123:0/1082078534 >> 172.21.15.123:0/1082078534 conn(0x7f46b00fee50 msgr2=0x7f46b0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 -- 172.21.15.123:0/1082078534 shutdown_connections 2024-03-24T00:38:32.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.604+0000 7f46b5000640 1 -- 172.21.15.123:0/1082078534 wait complete. 2024-03-24T00:38:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:32 smithi123 ceph-mon[31487]: pgmap v404: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-24T00:38:32.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.605+0000 7f46b5000640 1 Processor -- start 2024-03-24T00:38:32.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.605+0000 7f46b5000640 1 -- start start 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0105490 0x7f46b01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b010f560 con 0x7f46b0105490 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b010f3e0 con 0x7f46b0104ad0 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46b5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b010f6e0 con 0x7f46b01038e0 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46ade00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37386/0 (socket says 172.21.15.123:37386) 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46ade00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56086/0 (socket says 172.21.15.123:56086) 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46a5400640 1 -- 172.21.15.123:0/1339789961 learned_addr learned my addr 172.21.15.123:0/1339789961 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46ae800640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0105490 0x7f46b01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.606+0000 7f46ade00640 1 -- 172.21.15.123:0/1339789961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 msgr2=0x7f46b019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46ade00640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46ade00640 1 -- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0105490 msgr2=0x7f46b01a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46ade00640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0105490 0x7f46b01a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46ade00640 1 -- 172.21.15.123:0/1339789961 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b01a1850 con 0x7f46b01038e0 2024-03-24T00:38:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46a5400640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46ade00640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46a0037040 tx=0x7f46a0035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46a7400640 1 -- 172.21.15.123:0/1339789961 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0039040 con 0x7f46b01038e0 2024-03-24T00:38:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46a7400640 1 -- 172.21.15.123:0/1339789961 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46a0003970 con 0x7f46b01038e0 2024-03-24T00:38:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b01a1ae0 con 0x7f46b01038e0 2024-03-24T00:38:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46a7400640 1 -- 172.21.15.123:0/1339789961 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a00043c0 con 0x7f46b01038e0 2024-03-24T00:38:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.607+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b01a1ff0 con 0x7f46b01038e0 2024-03-24T00:38:32.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.610+0000 7f46a7400640 1 -- 172.21.15.123:0/1339789961 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f46a0046030 con 0x7f46b01038e0 2024-03-24T00:38:32.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.610+0000 7f46b7dc5740 1 Processor -- start 2024-03-24T00:38:32.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 -- start start 2024-03-24T00:38:32.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 0x55c596adf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adfa90 0x55c596ae4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c596a44eb0 con 0x55c596adfa90 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5969eda60 con 0x55c596ae4c00 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c596a40f50 con 0x55c596adf0a0 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46ade00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adfa90 0x55c596ae4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37390/0 (socket says 172.21.15.123:37390) 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46a5400640 1 -- 172.21.15.123:0/2261014811 learned_addr learned my addr 172.21.15.123:0/2261014811 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.611+0000 7f46ae800640 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 0x55c596adf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 -- 172.21.15.123:0/2261014811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 msgr2=0x55c596adf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 0x55c596adf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 -- 172.21.15.123:0/2261014811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adfa90 msgr2=0x55c596ae4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adfa90 0x55c596ae4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 -- 172.21.15.123:0/2261014811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5968217e0 con 0x55c596ae4c00 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46ae800640 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 0x55c596adf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:32.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a5400640 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4698011650 tx=0x7f4698009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2466a0db09fb38f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a7400640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 0x7f46940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f469f400640 1 -- 172.21.15.123:0/2261014811 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469801e070 con 0x55c596ae4c00 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f469f400640 1 -- 172.21.15.123:0/2261014811 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f469800fba0 con 0x55c596ae4c00 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f469f400640 1 -- 172.21.15.123:0/2261014811 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698019cd0 con 0x55c596ae4c00 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.612+0000 7f46a7400640 1 -- 172.21.15.123:0/1339789961 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f46a00f0640 con 0x7f46b01038e0 2024-03-24T00:38:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46ae800640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 0x7f46940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:32 smithi177 ceph-mon[36034]: pgmap v404: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46ae800640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 0x7f46940a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55c596ae02f0 tx=0x7f46a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2261014811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 msgr2=0x55c596ae7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4698011650 tx=0x7f4698009f90 comp rx=0 tx=0).stop 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2261014811 shutdown_connections 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae7090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adfa90 0x55c596ae4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2261014811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adf0a0 0x55c596adf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2261014811 >> 172.21.15.123:0/2261014811 conn(0x55c596ada610 msgr2=0x55c596adaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2261014811 shutdown_connections 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.613+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2261014811 wait complete. 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 -- start start 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 Processor -- start 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 0x55c596ae92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596ae97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c596ae7bb0 con 0x55c596ae4c00 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c596a00eb0 con 0x55c596adf0a0 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c596ae7d30 con 0x55c596adfa90 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.614+0000 7f46b7dc5740 10 client.0 _ll_register_callbacks cb 0x55c5969f5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ae800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 0x55c596ae92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ae800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35036/0 (socket says 172.21.15.123:35036) 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ae800640 1 -- 172.21.15.123:0/423792875 learned_addr learned my addr 172.21.15.123:0/423792875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46a5400640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596ae97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 -- 172.21.15.123:0/423792875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 msgr2=0x55c596ae97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596ae97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 -- 172.21.15.123:0/423792875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 msgr2=0x55c596af0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 -- 172.21.15.123:0/423792875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c596881f70 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ae800640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f46ade00640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 0x55c596ae92b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c596ae0500 tx=0x7f46a0035f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=301fc668abb7885e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f4693e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.615+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a00bcad0 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f46a00b9600 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f469800fba0 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46a00bc8f0 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 10 client.25346 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c596719170 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f469800fba0 con 0x55c596adf0a0 2024-03-24T00:38:32.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f46a003c030 con 0x55c596adf0a0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 10 client.25346 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 10 client.25346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 20 client.25346 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596af51f0 0x55c596af7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 4 client.25346 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.616+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c596a523f0 con 0x55c596af51f0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.617+0000 7f46a5400640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596af51f0 0x55c596af7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.617+0000 7f46a5400640 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596af51f0 0x55c596af7670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2496a101a82a349 server_cookie=da1295460bc56869 in_seq=0 out_seq=0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.617+0000 7f4693e00640 10 client.25346 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f4693e00640 1 -- 172.21.15.123:0/423792875 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c596a523f0 con 0x55c596af51f0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f4693e00640 10 client.25346 handle_command_reply: tid=0 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 client.25346 shutdown 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 msgr2=0x55c596ae92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 0x55c596ae92b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c596ae0500 tx=0x7f46a0035f00 comp rx=0 tx=0).stop 2024-03-24T00:38:32.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 shutdown_connections 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596ae4c00 0x55c596af0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596af51f0 0x55c596af7670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596ae97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/423792875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596adf0a0 0x55c596ae92b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 >> 172.21.15.123:0/423792875 conn(0x55c596ada610 msgr2=0x55c596ad9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f4693e00640 0 client.25346 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 shutdown_connections 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 1 -- 172.21.15.123:0/423792875 wait complete. 2024-03-24T00:38:32.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.618+0000 7f46b7dc5740 20 client.25346 trim_cache size 0 max 16384 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 Processor -- start 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 -- start start 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596ae9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596af07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01a8b30 con 0x55c596adf0a0 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01ab0f0 con 0x55c596ae4c00 2024-03-24T00:38:32.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01a9130 con 0x55c596adfa90 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46ade00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596ae9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37402/0 (socket says 172.21.15.123:37402) 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.636+0000 7f46a5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596af07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 -- 172.21.15.123:0/2368092759 learned_addr learned my addr 172.21.15.123:0/2368092759 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 -- 172.21.15.123:0/2368092759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 msgr2=0x55c596af07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596af07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 -- 172.21.15.123:0/2368092759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 msgr2=0x55c596ae9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596ae9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 -- 172.21.15.123:0/2368092759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c596ae0320 con 0x55c596ae4c00 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f46ae800640 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f46a801eb20 tx=0x7f46a801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85d61956bedceb54 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.637+0000 7f469ea00640 1 -- 172.21.15.123:0/2368092759 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8004020 con 0x55c596ae4c00 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f469ea00640 1 -- 172.21.15.123:0/2368092759 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46a80041c0 con 0x55c596ae4c00 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f469ea00640 1 -- 172.21.15.123:0/2368092759 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a800f700 con 0x55c596ae4c00 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2368092759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 msgr2=0x55c596ae0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f46a801eb20 tx=0x7f46a801b000 comp rx=0 tx=0).stop 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2368092759 shutdown_connections 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596ae0530 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596af07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/2368092759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596ae9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2368092759 >> 172.21.15.123:0/2368092759 conn(0x55c596ada610 msgr2=0x55c596ade100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2368092759 shutdown_connections 2024-03-24T00:38:32.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.638+0000 7f46b7dc5740 1 -- 172.21.15.123:0/2368092759 wait complete. 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 -- start start 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 Processor -- start 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596aeec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01a9770 con 0x55c596adf0a0 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01aaa90 con 0x55c596ae4c00 2024-03-24T00:38:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b01a9dd0 con 0x55c596adfa90 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46b7dc5740 10 client.0 _ll_register_callbacks cb 0x7f46a0009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.639+0000 7f46ade00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ade00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35054/0 (socket says 172.21.15.123:35054) 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ade00640 1 -- 172.21.15.123:0/1852597469 learned_addr learned my addr 172.21.15.123:0/1852597469 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37410/0 (socket says 172.21.15.123:37410) 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46a5400640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596aeec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 -- 172.21.15.123:0/1852597469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 msgr2=0x55c596aeec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596aeec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 -- 172.21.15.123:0/1852597469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 msgr2=0x55c596aee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 -- 172.21.15.123:0/1852597469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c596af0210 con 0x55c596ae4c00 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ade00640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46ae800640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f46a801eaf0 tx=0x7f46a8003ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e4472a566748719 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f4693e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8037020 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f46a8005050 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.640+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46a80041c0 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46a8003fe0 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 10 client.25352 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c596b59b50 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46a80041c0 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f46a8004b30 con 0x55c596ae4c00 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 10 client.25352 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 10 client.25352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 20 client.25352 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596b5a450 0x55c596b5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 4 client.25352 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c596a4c740 con 0x55c596b5a450 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.641+0000 7f46ade00640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596b5a450 0x55c596b5c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.642+0000 7f46ade00640 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596b5a450 0x55c596b5c870 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b21b6d2184f7e6 server_cookie=9372cd93de648b92 in_seq=0 out_seq=0 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.642+0000 7f4693e00640 10 client.25352 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f4693e00640 1 -- 172.21.15.123:0/1852597469 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c596a4c740 con 0x55c596b5a450 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f4693e00640 10 client.25352 handle_command_reply: tid=0 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 client.25352 shutdown 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 msgr2=0x55c596aefc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f46a801eaf0 tx=0x7f46a8003ad0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 shutdown_connections 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c596ae4c00 0x55c596aefc20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c596b5a450 0x55c596b5c870 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c596adfa90 0x55c596aeec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 --2- 172.21.15.123:0/1852597469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c596adf0a0 0x55c596aee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f4693e00640 0 client.25352 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 >> 172.21.15.123:0/1852597469 conn(0x55c596ada610 msgr2=0x55c596ade100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 shutdown_connections 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 1 -- 172.21.15.123:0/1852597469 wait complete. 2024-03-24T00:38:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.643+0000 7f46b7dc5740 20 client.25352 trim_cache size 0 max 16384 2024-03-24T00:38:32.648 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:38:32.648 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:38:32.648 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "ae193bd9-744e-4b3d-8986-6d6ad70aa53c", 2024-03-24T00:38:32.648 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:38:32.648 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 msgr2=0x7f46940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 0x7f46940a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55c596ae02f0 tx=0x7f46a8005f90 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 msgr2=0x7f46b0199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46a0037040 tx=0x7f46a0035d10 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 shutdown_connections 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46940a2880 0x7f46940a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46b0105490 0x7f46b01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46b0104ad0 0x7f46b019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 --2- 172.21.15.123:0/1339789961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46b01038e0 0x7f46b0199c40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 >> 172.21.15.123:0/1339789961 conn(0x7f46b00fee50 msgr2=0x7f46b0107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 shutdown_connections 2024-03-24T00:38:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.644+0000 7f46b5000640 1 -- 172.21.15.123:0/1339789961 wait complete. 2024-03-24T00:38:32.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:38:32.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 -- 172.21.15.123:0/524610988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8170077c40 msgr2=0x7f81700784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 --2- 172.21.15.123:0/524610988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8170077c40 0x7f81700784e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8168009ca0 tx=0x7f816802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 -- 172.21.15.123:0/524610988 shutdown_connections 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 --2- 172.21.15.123:0/524610988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8170077c40 0x7f81700784e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 --2- 172.21.15.123:0/524610988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81700772a0 0x7f8170077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 --2- 172.21.15.123:0/524610988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170076060 0x7f8170076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.859+0000 7f8177400640 1 -- 172.21.15.123:0/524610988 >> 172.21.15.123:0/524610988 conn(0x7f81700fee50 msgr2=0x7f8170101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.860+0000 7f8177400640 1 -- 172.21.15.123:0/524610988 shutdown_connections 2024-03-24T00:38:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.860+0000 7f8177400640 1 -- 172.21.15.123:0/524610988 wait complete. 2024-03-24T00:38:32.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.860+0000 7f8177400640 1 Processor -- start 2024-03-24T00:38:32.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 -- start start 2024-03-24T00:38:32.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170076060 0x7f817019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170077c40 0x7f81701a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8170113950 con 0x7f8170077c40 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701137d0 con 0x7f81700772a0 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8177400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8170113ad0 con 0x7f8170076060 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8167e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8167e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37414/0 (socket says 172.21.15.123:37414) 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8167e00640 1 -- 172.21.15.123:0/2369928741 learned_addr learned my addr 172.21.15.123:0/2369928741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8175400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170077c40 0x7f81701a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.861+0000 7f8174a00640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170076060 0x7f817019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 -- 172.21.15.123:0/2369928741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170076060 msgr2=0x7f817019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170076060 0x7f817019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 -- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170077c40 msgr2=0x7f81701a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170077c40 0x7f81701a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 -- 172.21.15.123:0/2369928741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81701a5ba0 con 0x7f81700772a0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8167e00640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9f0 tx=0x7f815800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f816e000640 1 -- 172.21.15.123:0/2369928741 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815800cd80 con 0x7f81700772a0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f816e000640 1 -- 172.21.15.123:0/2369928741 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8158005500 con 0x7f81700772a0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81701a5e90 con 0x7f81700772a0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f816e000640 1 -- 172.21.15.123:0/2369928741 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158010690 con 0x7f81700772a0 2024-03-24T00:38:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.862+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81701a6350 con 0x7f81700772a0 2024-03-24T00:38:32.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.865+0000 7f816e000640 1 -- 172.21.15.123:0/2369928741 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f81580046c0 con 0x7f81700772a0 2024-03-24T00:38:32.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.865+0000 7f817a001740 1 Processor -- start 2024-03-24T00:38:32.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 -- start start 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed399122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed39919e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed39877250 con 0x55ed39911e80 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3981fa30 con 0x55ed39912870 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f817a001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed398732f0 con 0x55ed399179e0 2024-03-24T00:38:32.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f8174a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f8174a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37426/0 (socket says 172.21.15.123:37426) 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f8174a00640 1 -- 172.21.15.123:0/3558371489 learned_addr learned my addr 172.21.15.123:0/3558371489 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f8175400640 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed399122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.866+0000 7f8167e00640 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed39919e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 -- 172.21.15.123:0/3558371489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 msgr2=0x55ed39919e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed39919e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 -- 172.21.15.123:0/3558371489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 msgr2=0x55ed399122a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed399122a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 -- 172.21.15.123:0/3558371489 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed3980a690 con 0x55ed39912870 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f816e000640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 0x7f81540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8174a00640 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8160009cf0 tx=0x7f8160011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab029421c0cc801 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f816e000640 1 -- 172.21.15.123:0/2369928741 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f81580cee20 con 0x7f81700772a0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8175400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 0x7f81540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8165c00640 1 -- 172.21.15.123:0/3558371489 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160004230 con 0x55ed39912870 2024-03-24T00:38:32.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8165c00640 1 -- 172.21.15.123:0/3558371489 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81600043d0 con 0x55ed39912870 2024-03-24T00:38:32.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.867+0000 7f8165c00640 1 -- 172.21.15.123:0/3558371489 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160019960 con 0x55ed39912870 2024-03-24T00:38:32.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 -- 172.21.15.123:0/3558371489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 msgr2=0x55ed39917370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8160009cf0 tx=0x7f8160011810 comp rx=0 tx=0).stop 2024-03-24T00:38:32.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f8175400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 0x7f81540a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f816802c630 tx=0x7f8168032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 -- 172.21.15.123:0/3558371489 shutdown_connections 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed39919e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39917370 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 --2- 172.21.15.123:0/3558371489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed399122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 -- 172.21.15.123:0/3558371489 >> 172.21.15.123:0/3558371489 conn(0x55ed398da200 msgr2=0x55ed3990d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 -- 172.21.15.123:0/3558371489 shutdown_connections 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.868+0000 7f817a001740 1 -- 172.21.15.123:0/3558371489 wait complete. 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 -- start start 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 Processor -- start 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39911e80 0x55ed3991be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 0x55ed3991c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed399233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3991a990 con 0x55ed39912870 2024-03-24T00:38:32.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed39832e80 con 0x55ed399179e0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3991ab10 con 0x55ed39911e80 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.869+0000 7f817a001740 10 client.0 _ll_register_callbacks cb 0x55ed39845130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 0x55ed3991c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8175400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed399233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8174a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39911e80 0x55ed3991be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8175400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed399233a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37430/0 (socket says 172.21.15.123:37430) 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8175400640 1 -- 172.21.15.123:0/265872639 learned_addr learned my addr 172.21.15.123:0/265872639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 -- 172.21.15.123:0/265872639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39911e80 msgr2=0x55ed3991be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39911e80 0x55ed3991be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 -- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 msgr2=0x55ed399233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed399233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 -- 172.21.15.123:0/265872639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed396557d0 con 0x55ed39912870 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f8167e00640 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 0x55ed3991c360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f815809b700 tx=0x7f815809b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5afc6e5979a4a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f815e000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:38:32.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815809b770 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8158097de0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.870+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f81600043d0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f815809f8c0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 10 client.15261 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ed3958b3f0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81600043d0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f815809abc0 con 0x55ed39912870 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 10 client.15261 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 10 client.15261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 20 client.15261 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed39927c60 0x55ed3992a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 4 client.15261 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.871+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed39884790 con 0x55ed39927c60 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.872+0000 7f8174a00640 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed39927c60 0x55ed3992a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.872+0000 7f8174a00640 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed39927c60 0x55ed3992a0b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b5b86045e09ba93 server_cookie=17d175fe11ed6f81 in_seq=0 out_seq=0 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.872+0000 7f815e000640 10 client.15261 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f815e000640 1 -- 172.21.15.123:0/265872639 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed39884790 con 0x55ed39927c60 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f815e000640 10 client.15261 handle_command_reply: tid=0 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 client.15261 shutdown 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 msgr2=0x55ed3991c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 0x55ed3991c360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f815809b700 tx=0x7f815809b600 comp rx=0 tx=0).stop 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 shutdown_connections 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed399233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39912870 0x55ed3991c360 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed39927c60 0x55ed3992a0b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 --2- 172.21.15.123:0/265872639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39911e80 0x55ed3991be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 >> 172.21.15.123:0/265872639 conn(0x55ed398da200 msgr2=0x55ed3990ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.873+0000 7f815e000640 0 client.15261 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.874+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 shutdown_connections 2024-03-24T00:38:32.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.874+0000 7f817a001740 1 -- 172.21.15.123:0/265872639 wait complete. 2024-03-24T00:38:32.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.874+0000 7f817a001740 20 client.15261 trim_cache size 0 max 16384 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 Processor -- start 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 -- start start 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39912870 0x55ed39923360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed39910f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701ad0c0 con 0x55ed39911e80 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701af680 con 0x55ed399179e0 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.891+0000 7f817a001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701ad6c0 con 0x55ed39912870 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35122/0 (socket says 172.21.15.123:35122) 2024-03-24T00:38:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 -- 172.21.15.123:0/3169609097 learned_addr learned my addr 172.21.15.123:0/3169609097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8167e00640 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39912870 0x55ed39923360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8175400640 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed39910f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 -- 172.21.15.123:0/3169609097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39912870 msgr2=0x55ed39923360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39912870 0x55ed39923360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 -- 172.21.15.123:0/3169609097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 msgr2=0x55ed39910f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed39910f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 -- 172.21.15.123:0/3169609097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed39917730 con 0x55ed39911e80 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.892+0000 7f8174a00640 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8160009230 tx=0x7f8160005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2c0af74458e41a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f8165200640 1 -- 172.21.15.123:0/3169609097 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160031070 con 0x55ed39911e80 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f8165200640 1 -- 172.21.15.123:0/3169609097 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8160005500 con 0x55ed39911e80 2024-03-24T00:38:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f8165200640 1 -- 172.21.15.123:0/3169609097 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816002b780 con 0x55ed39911e80 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f817a001740 1 -- 172.21.15.123:0/3169609097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 msgr2=0x55ed39910a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f817a001740 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8160009230 tx=0x7f8160005850 comp rx=0 tx=0).stop 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f817a001740 1 -- 172.21.15.123:0/3169609097 shutdown_connections 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f817a001740 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed399179e0 0x55ed39910f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.893+0000 7f817a001740 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed39912870 0x55ed39923360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 --2- 172.21.15.123:0/3169609097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39910a30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 -- 172.21.15.123:0/3169609097 >> 172.21.15.123:0/3169609097 conn(0x55ed398da200 msgr2=0x55ed39913310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 -- 172.21.15.123:0/3169609097 shutdown_connections 2024-03-24T00:38:32.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 -- 172.21.15.123:0/3169609097 wait complete. 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 -- start start 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 Processor -- start 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.894+0000 7f817a001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39921960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f817a001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed399228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f817a001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701add00 con 0x55ed39911e80 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f817a001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701af020 con 0x55ed39912870 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f817a001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701ae360 con 0x55ed399179e0 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f817a001740 10 client.0 _ll_register_callbacks cb 0x7f815800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35124/0 (socket says 172.21.15.123:35124) 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 -- 172.21.15.123:0/3515244971 learned_addr learned my addr 172.21.15.123:0/3515244971 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8167e00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39921960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8175400640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed399228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 -- 172.21.15.123:0/3515244971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 msgr2=0x55ed399228b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed399228b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 -- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 msgr2=0x55ed39921960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39921960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 -- 172.21.15.123:0/3515244971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed3954b0f0 con 0x55ed39911e80 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8167e00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39921960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.895+0000 7f8174a00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f816001b040 tx=0x7f8160011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0c45792f1a5287 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160039060 con 0x55ed39911e80 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8160005500 con 0x55ed39911e80 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8160003800 con 0x55ed39911e80 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8160031070 con 0x55ed39911e80 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f817a001740 10 client.15267 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ed39983c30 con 0x55ed39911e80 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8160005500 con 0x55ed39911e80 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.896+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f816002b070 con 0x55ed39911e80 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 10 client.15267 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 10 client.15267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 20 client.15267 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed399843a0 0x55ed399867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 4 client.15267 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed3987eae0 con 0x55ed399843a0 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f8167e00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed399843a0 0x55ed399867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f8167e00640 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed399843a0 0x55ed399867c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389314531deb57f8 server_cookie=c6c0c8d39a81a3ff in_seq=0 out_seq=0 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f815e000640 10 client.15267 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f815e000640 1 -- 172.21.15.123:0/3515244971 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ed3987eae0 con 0x55ed399843a0 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f815e000640 10 client.15267 handle_command_reply: tid=0 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.897+0000 7f817a001740 1 client.15267 shutdown 2024-03-24T00:38:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 msgr2=0x55ed39921420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f816001b040 tx=0x7f8160011e50 comp rx=0 tx=0).stop 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 shutdown_connections 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ed399843a0 0x55ed399867c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed399179e0 0x55ed399228b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed39912870 0x55ed39921960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 --2- 172.21.15.123:0/3515244971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ed39911e80 0x55ed39921420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 >> 172.21.15.123:0/3515244971 conn(0x55ed398da200 msgr2=0x55ed3990c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f815e000640 0 client.15267 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 shutdown_connections 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 1 -- 172.21.15.123:0/3515244971 wait complete. 2024-03-24T00:38:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.898+0000 7f817a001740 20 client.15267 trim_cache size 0 max 16384 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "ae193bd9-744e-4b3d-8986-6d6ad70aa53c": { 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "ae193bd9-744e-4b3d-8986-6d6ad70aa53c", 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:38:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 msgr2=0x7f81540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 0x7f81540a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f816802c630 tx=0x7f8168032590 comp rx=0 tx=0).stop 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 msgr2=0x7f817019e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9f0 tx=0x7f815800edf0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 shutdown_connections 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f81540a2880 0x7f81540a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170077c40 0x7f81701a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81700772a0 0x7f817019e4c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 --2- 172.21.15.123:0/2369928741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170076060 0x7f817019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 >> 172.21.15.123:0/2369928741 conn(0x7f81700fee50 msgr2=0x7f8170100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 shutdown_connections 2024-03-24T00:38:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:38:32.899+0000 7f8177400640 1 -- 172.21.15.123:0/2369928741 wait complete. 2024-03-24T00:38:32.918 INFO:tasks.cephfs.filesystem:scrub status for tag:ae193bd9-744e-4b3d-8986-6d6ad70aa53c - {'path': '/', 'tag': 'ae193bd9-744e-4b3d-8986-6d6ad70aa53c', 'options': 'recursive,force'} 2024-03-24T00:38:33.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:33 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:38:33.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:33 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:33.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:33 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:38:33.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:33 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:33 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:38:33.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:33 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:33.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:33 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:38:33.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:33 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:33 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:38:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:33 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:33 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:38:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:33 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:38:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:34 smithi204 ceph-mon[36345]: pgmap v405: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T00:38:34.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:34 smithi123 ceph-mon[31487]: pgmap v405: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T00:38:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:34 smithi177 ceph-mon[36034]: pgmap v405: 97 pgs: 97 active+clean; 653 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T00:38:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:36 smithi204 ceph-mon[36345]: pgmap v406: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:36.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:36 smithi123 ceph-mon[31487]: pgmap v406: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:36 smithi177 ceph-mon[36034]: pgmap v406: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T00:38:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:37 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:38:37.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:37 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:38:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:37 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:38:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:38 smithi204 ceph-mon[36345]: pgmap v407: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:38 smithi123 ceph-mon[31487]: pgmap v407: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:38 smithi177 ceph-mon[36034]: pgmap v407: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.355+0000 7fac8f200640 1 -- 172.21.15.123:0/240647721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90102710 msgr2=0x7fac90102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.355+0000 7fac8f200640 1 --2- 172.21.15.123:0/240647721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90102710 0x7fac90102b90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009900 tx=0x7fac7c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 -- 172.21.15.123:0/240647721 shutdown_connections 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 --2- 172.21.15.123:0/240647721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 0x7fac90109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 --2- 172.21.15.123:0/240647721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90102710 0x7fac90102b90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 --2- 172.21.15.123:0/240647721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90101520 0x7fac90101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 -- 172.21.15.123:0/240647721 >> 172.21.15.123:0/240647721 conn(0x7fac900fcab0 msgr2=0x7fac900feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 -- 172.21.15.123:0/240647721 shutdown_connections 2024-03-24T00:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.356+0000 7fac8f200640 1 -- 172.21.15.123:0/240647721 wait complete. 2024-03-24T00:38:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 Processor -- start 2024-03-24T00:38:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 -- start start 2024-03-24T00:38:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90101520 0x7fac90199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 0x7fac901a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9010d200 con 0x7fac90101520 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9010d080 con 0x7fac90102710 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9010d380 con 0x7fac901030d0 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46422/0 (socket says 172.21.15.123:46422) 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90101520 0x7fac90199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8e800640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 0x7fac901a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.357+0000 7fac8d400640 1 -- 172.21.15.123:0/1810045835 learned_addr learned my addr 172.21.15.123:0/1810045835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 -- 172.21.15.123:0/1810045835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 msgr2=0x7fac901a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 0x7fac901a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 -- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90101520 msgr2=0x7fac90199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90101520 0x7fac90199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 -- 172.21.15.123:0/1810045835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac901a1620 con 0x7fac90102710 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8d400640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac7c005ab0 tx=0x7fac7c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac86a00640 1 -- 172.21.15.123:0/1810045835 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c043020 con 0x7fac90102710 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac86a00640 1 -- 172.21.15.123:0/1810045835 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac7c0054e0 con 0x7fac90102710 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.358+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac901a18b0 con 0x7fac90102710 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.359+0000 7fac86a00640 1 -- 172.21.15.123:0/1810045835 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c003ce0 con 0x7fac90102710 2024-03-24T00:38:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.359+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac901a1e00 con 0x7fac90102710 2024-03-24T00:38:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.362+0000 7fac86a00640 1 -- 172.21.15.123:0/1810045835 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fac7c046020 con 0x7fac90102710 2024-03-24T00:38:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.362+0000 7fac97a3e740 1 Processor -- start 2024-03-24T00:38:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.362+0000 7fac97a3e740 1 -- start start 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x5627473168e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473193e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627472d43b0 con 0x562747316f50 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562747276830 con 0x5627473113f0 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac97a3e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274721f5d0 con 0x562747311de0 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x5627473168e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46436/0 (socket says 172.21.15.123:46436) 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x5627473168e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:36110/0 (socket says 172.21.15.123:36110) 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 -- 172.21.15.123:0/190615164 learned_addr learned my addr 172.21.15.123:0/190615164 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8d400640 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 -- 172.21.15.123:0/190615164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 msgr2=0x5627473168e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x5627473168e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 -- 172.21.15.123:0/190615164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 msgr2=0x5627473193e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473193e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.363+0000 7fac8e800640 1 -- 172.21.15.123:0/190615164 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627472372c0 con 0x5627473113f0 2024-03-24T00:38:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac8e800640 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fac88010370 tx=0x7fac88010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e1555ef9d271513 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac86a00640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 0x7fac740a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac86a00640 1 -- 172.21.15.123:0/1810045835 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fac7c0f0b40 con 0x7fac90102710 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac7b400640 1 -- 172.21.15.123:0/190615164 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88016070 con 0x5627473113f0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac7b400640 1 -- 172.21.15.123:0/190615164 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac88004d20 con 0x5627473113f0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac8de00640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 0x7fac740a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.364+0000 7fac7b400640 1 -- 172.21.15.123:0/190615164 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88005480 con 0x5627473113f0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 -- 172.21.15.123:0/190615164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 msgr2=0x562747311810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fac88010370 tx=0x7fac88010830 comp rx=0 tx=0).stop 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac8de00640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 0x7fac740a4cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5627473121e0 tx=0x7fac80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 -- 172.21.15.123:0/190615164 shutdown_connections 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473193e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x5627473168e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 --2- 172.21.15.123:0/190615164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x562747311810 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 -- 172.21.15.123:0/190615164 >> 172.21.15.123:0/190615164 conn(0x56274730c960 msgr2=0x56274730cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 -- 172.21.15.123:0/190615164 shutdown_connections 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.365+0000 7fac97a3e740 1 -- 172.21.15.123:0/190615164 wait complete. 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.366+0000 7fac97a3e740 1 -- start start 2024-03-24T00:38:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.366+0000 7fac97a3e740 1 Processor -- start 2024-03-24T00:38:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x56274731bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x562747322b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562747232a20 con 0x562747316f50 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627472728d0 con 0x5627473113f0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562747319f00 con 0x562747311de0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac97a3e740 10 client.0 _ll_register_callbacks cb 0x56274726f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46442/0 (socket says 172.21.15.123:46442) 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 -- 172.21.15.123:0/3932052413 learned_addr learned my addr 172.21.15.123:0/3932052413 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8e800640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x562747322b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8d400640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x56274731bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 -- 172.21.15.123:0/3932052413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 msgr2=0x56274731bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x56274731bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 -- 172.21.15.123:0/3932052413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 msgr2=0x562747322b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x562747322b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.367+0000 7fac8de00640 1 -- 172.21.15.123:0/3932052413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562746f8b210 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac8de00640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fac800269d0 tx=0x7fac80004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c5759e5644734d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac80036060 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fac80020030 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac88004d20 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac80009dc0 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac97a3e740 10 client.25382 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.368+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562747057620 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac88004d20 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fac800246a0 con 0x5627473113f0 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 10 client.25382 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 10 client.25382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 20 client.25382 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747327460 0x562747329850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 4 client.25382 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562747283d70 con 0x562747327460 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.369+0000 7fac8d400640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747327460 0x562747329850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.370+0000 7fac8d400640 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747327460 0x562747329850 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31cfcee1bd5013 server_cookie=594d002837582768 in_seq=0 out_seq=0 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.370+0000 7fac6fe00640 10 client.25382 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.370+0000 7fac6fe00640 1 -- 172.21.15.123:0/3932052413 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562747283d70 con 0x562747327460 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.370+0000 7fac6fe00640 10 client.25382 handle_command_reply: tid=0 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.370+0000 7fac97a3e740 1 client.25382 shutdown 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 msgr2=0x56274731b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fac800269d0 tx=0x7fac80004730 comp rx=0 tx=0).stop 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 shutdown_connections 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x562747322b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747327460 0x562747329850 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562747311de0 0x56274731bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 --2- 172.21.15.123:0/3932052413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5627473113f0 0x56274731b600 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac6fe00640 0 client.25382 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 >> 172.21.15.123:0/3932052413 conn(0x56274730c960 msgr2=0x56274730bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 shutdown_connections 2024-03-24T00:38:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 1 -- 172.21.15.123:0/3932052413 wait complete. 2024-03-24T00:38:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.371+0000 7fac97a3e740 20 client.25382 trim_cache size 0 max 16384 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 Processor -- start 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 -- start start 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x56274731b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x56274730b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a47b0 con 0x562747316f50 2024-03-24T00:38:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a6d30 con 0x562747311de0 2024-03-24T00:38:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.386+0000 7fac97a3e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a4db0 con 0x5627473113f0 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x56274730b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46458/0 (socket says 172.21.15.123:46458) 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 -- 172.21.15.123:0/313117915 learned_addr learned my addr 172.21.15.123:0/313117915 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8de00640 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x56274731b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 -- 172.21.15.123:0/313117915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 msgr2=0x56274731b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x56274731b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 -- 172.21.15.123:0/313117915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 msgr2=0x56274730b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x56274730b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 -- 172.21.15.123:0/313117915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627473117d0 con 0x562747311de0 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8de00640 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x56274731b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.387+0000 7fac8d400640 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fac7c006010 tx=0x7fac7c041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7a8a34862501609 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac7aa00640 1 -- 172.21.15.123:0/313117915 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0d5070 con 0x562747311de0 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac7aa00640 1 -- 172.21.15.123:0/313117915 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac7c004c80 con 0x562747311de0 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac7aa00640 1 -- 172.21.15.123:0/313117915 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0bd4f0 con 0x562747311de0 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 -- 172.21.15.123:0/313117915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 msgr2=0x562747322b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fac7c006010 tx=0x7fac7c041040 comp rx=0 tx=0).stop 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 -- 172.21.15.123:0/313117915 shutdown_connections 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x56274730b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747322b10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 --2- 172.21.15.123:0/313117915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x56274731b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.388+0000 7fac97a3e740 1 -- 172.21.15.123:0/313117915 >> 172.21.15.123:0/313117915 conn(0x56274730c960 msgr2=0x56274731b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- 172.21.15.123:0/313117915 shutdown_connections 2024-03-24T00:38:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- 172.21.15.123:0/313117915 wait complete. 2024-03-24T00:38:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- start start 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 Processor -- start 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x5627473204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473219a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a53b0 con 0x562747316f50 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a66d0 con 0x562747311de0 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.389+0000 7fac97a3e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac901a5a10 con 0x5627473113f0 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac97a3e740 10 client.0 _ll_register_callbacks cb 0x7fac80009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46466/0 (socket says 172.21.15.123:46466) 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 -- 172.21.15.123:0/629227118 learned_addr learned my addr 172.21.15.123:0/629227118 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8e800640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473219a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8de00640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x5627473204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 -- 172.21.15.123:0/629227118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 msgr2=0x5627473204f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x5627473204f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 -- 172.21.15.123:0/629227118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 msgr2=0x5627473219a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473219a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 -- 172.21.15.123:0/629227118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562747321f90 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8de00640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x5627473204f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac8d400640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0bd1f0 tx=0x7fac7c0bdcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2c2628c84a6fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.390+0000 7fac6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0d5070 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fac7c0bb590 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac7c004c80 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac7c0bce50 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 10 client.25388 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562747382e20 con 0x562747311de0 2024-03-24T00:38:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac7c004c80 con 0x562747311de0 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fac7c0b9b60 con 0x562747311de0 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 10 client.25388 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 10 client.25388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 20 client.25388 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747383640 0x562747385ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 4 client.25388 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.391+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56274727e0c0 con 0x562747383640 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.392+0000 7fac8de00640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747383640 0x562747385ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.392+0000 7fac8de00640 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747383640 0x562747385ac0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12eba66a0e46adb3 server_cookie=803d16cfd4571590 in_seq=0 out_seq=0 2024-03-24T00:38:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.392+0000 7fac6fe00640 10 client.25388 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.398+0000 7fac6fe00640 1 -- 172.21.15.123:0/629227118 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56274727e0c0 con 0x562747383640 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.398+0000 7fac6fe00640 10 client.25388 handle_command_reply: tid=0 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 client.25388 shutdown 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 msgr2=0x562747320a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0bd1f0 tx=0x7fac7c0bdcd0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 shutdown_connections 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562747316f50 0x5627473219a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562747311de0 0x562747320a30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562747383640 0x562747385ac0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 --2- 172.21.15.123:0/629227118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627473113f0 0x5627473204f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac6fe00640 0 client.25388 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 >> 172.21.15.123:0/629227118 conn(0x56274730c960 msgr2=0x56274730fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 shutdown_connections 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 1 -- 172.21.15.123:0/629227118 wait complete. 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.399+0000 7fac97a3e740 20 client.25388 trim_cache size 0 max 16384 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:38:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 msgr2=0x7fac740a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 0x7fac740a4cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5627473121e0 tx=0x7fac80005f90 comp rx=0 tx=0).stop 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 msgr2=0x7fac90199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac7c005ab0 tx=0x7fac7c02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 shutdown_connections 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac901030d0 0x7fac901a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac90102710 0x7fac90199f40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fac740a27f0 0x7fac740a4cb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 --2- 172.21.15.123:0/1810045835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac90101520 0x7fac90199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 >> 172.21.15.123:0/1810045835 conn(0x7fac900fcab0 msgr2=0x7fac900738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 shutdown_connections 2024-03-24T00:38:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:39.400+0000 7fac8f200640 1 -- 172.21.15.123:0/1810045835 wait complete. 2024-03-24T00:38:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:40 smithi204 ceph-mon[36345]: pgmap v408: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T00:38:40.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:40 smithi123 ceph-mon[31487]: pgmap v408: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T00:38:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:40 smithi177 ceph-mon[36034]: pgmap v408: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T00:38:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:42 smithi204 ceph-mon[36345]: pgmap v409: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:42 smithi123 ceph-mon[31487]: pgmap v409: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:42 smithi177 ceph-mon[36034]: pgmap v409: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:44 smithi204 ceph-mon[36345]: pgmap v410: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-24T00:38:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:44 smithi123 ceph-mon[31487]: pgmap v410: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-24T00:38:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:44 smithi177 ceph-mon[36034]: pgmap v410: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-24T00:38:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:46 smithi204 ceph-mon[36345]: pgmap v411: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:46 smithi123 ceph-mon[31487]: pgmap v411: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:46 smithi177 ceph-mon[36034]: pgmap v411: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T00:38:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:48 smithi204 ceph-mon[36345]: pgmap v412: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:48 smithi123 ceph-mon[31487]: pgmap v412: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:48 smithi177 ceph-mon[36034]: pgmap v412: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:50 smithi204 ceph-mon[36345]: pgmap v413: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:50 smithi123 ceph-mon[31487]: pgmap v413: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:50 smithi177 ceph-mon[36034]: pgmap v413: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T00:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 -- 172.21.15.123:0/955008349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 msgr2=0x7eff08102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 --2- 172.21.15.123:0/955008349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08102d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efef4009900 tx=0x7efef402bc30 comp rx=0 tx=0).stop 2024-03-24T00:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 -- 172.21.15.123:0/955008349 shutdown_connections 2024-03-24T00:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 --2- 172.21.15.123:0/955008349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff08103280 0x7eff08109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 --2- 172.21.15.123:0/955008349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08102d40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 --2- 172.21.15.123:0/955008349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff081016d0 0x7eff08101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 -- 172.21.15.123:0/955008349 >> 172.21.15.123:0/955008349 conn(0x7eff080fcc60 msgr2=0x7eff080ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 -- 172.21.15.123:0/955008349 shutdown_connections 2024-03-24T00:38:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.545+0000 7eff0de00640 1 -- 172.21.15.123:0/955008349 wait complete. 2024-03-24T00:38:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 Processor -- start 2024-03-24T00:38:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 -- start start 2024-03-24T00:38:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff081016d0 0x7eff08197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 0x7eff0819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810d380 con 0x7eff081016d0 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810d200 con 0x7eff081028c0 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.547+0000 7eff0de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810d500 con 0x7eff08103280 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff081016d0 0x7eff08197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39814/0 (socket says 172.21.15.123:39814) 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 -- 172.21.15.123:0/3579033245 learned_addr learned my addr 172.21.15.123:0/3579033245 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff07400640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 0x7eff0819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 -- 172.21.15.123:0/3579033245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 msgr2=0x7eff0819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 0x7eff0819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 -- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff081016d0 msgr2=0x7eff08197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff081016d0 0x7eff08197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 -- 172.21.15.123:0/3579033245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0819f520 con 0x7eff081028c0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff07400640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 0x7eff0819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.548+0000 7eff06000640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efef4005ab0 tx=0x7efef402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.549+0000 7eff0ca00640 1 -- 172.21.15.123:0/3579033245 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4043020 con 0x7eff081028c0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.549+0000 7eff0ca00640 1 -- 172.21.15.123:0/3579033245 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efef40054e0 con 0x7eff081028c0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.549+0000 7eff0ca00640 1 -- 172.21.15.123:0/3579033245 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4003ce0 con 0x7eff081028c0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.549+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0819f7b0 con 0x7eff081028c0 2024-03-24T00:38:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.549+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0819fcb0 con 0x7eff081028c0 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 Processor -- start 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 -- start start 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 0x565091b19ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b1a150 0x565091b1c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091ad33b0 con 0x565091b14fe0 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.552+0000 7eff10bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091a75830 con 0x565091b145f0 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff10bf5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091a1e5d0 con 0x565091b1a150 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39830/0 (socket says 172.21.15.123:39830) 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 -- 172.21.15.123:0/309078067 learned_addr learned my addr 172.21.15.123:0/309078067 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff06a00640 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 0x565091b19ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 -- 172.21.15.123:0/309078067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b1a150 msgr2=0x565091b1c5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b1a150 0x565091b1c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 -- 172.21.15.123:0/309078067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 msgr2=0x565091b19ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 0x565091b19ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 -- 172.21.15.123:0/309078067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565091a362c0 con 0x565091b145f0 2024-03-24T00:38:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff06a00640 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 0x565091b19ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff0ca00640 1 -- 172.21.15.123:0/3579033245 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efef4046020 con 0x7eff081028c0 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.553+0000 7eff07400640 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff00009f10 tx=0x7eff0000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2396d7d47a45751 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.554+0000 7efef3e00640 1 -- 172.21.15.123:0/309078067 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00003b70 con 0x565091b145f0 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.554+0000 7efef3e00640 1 -- 172.21.15.123:0/309078067 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff00003d10 con 0x565091b145f0 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.554+0000 7efef3e00640 1 -- 172.21.15.123:0/309078067 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00011a70 con 0x565091b145f0 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 -- 172.21.15.123:0/309078067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 msgr2=0x565091b14a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff00009f10 tx=0x7eff0000f730 comp rx=0 tx=0).stop 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 -- 172.21.15.123:0/309078067 shutdown_connections 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b1a150 0x565091b1c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b14fe0 0x565091b19ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 --2- 172.21.15.123:0/309078067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b145f0 0x565091b14a10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff0ca00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 0x7efeec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff10bf5740 1 -- 172.21.15.123:0/309078067 >> 172.21.15.123:0/309078067 conn(0x565091b0b750 msgr2=0x565091b0bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff0ca00640 1 -- 172.21.15.123:0/3579033245 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efef40f0b80 con 0x7eff081028c0 2024-03-24T00:38:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.555+0000 7eff07400640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 0x7efeec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.556+0000 7eff10bf5740 1 -- 172.21.15.123:0/309078067 shutdown_connections 2024-03-24T00:38:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.556+0000 7eff10bf5740 1 -- 172.21.15.123:0/309078067 wait complete. 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.556+0000 7eff10bf5740 1 -- start start 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.556+0000 7eff07400640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 0x7efeec0a4de0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff00011580 tx=0x7eff00009aa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 Processor -- start 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b1e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b1ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091a31a20 con 0x565091b145f0 2024-03-24T00:38:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091a718d0 con 0x565091b1a150 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565091b1d100 con 0x565091b14fe0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.557+0000 7eff10bf5740 10 client.0 _ll_register_callbacks cb 0x565091a6e1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff06a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b1e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39840/0 (socket says 172.21.15.123:39840) 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 -- 172.21.15.123:0/3432763721 learned_addr learned my addr 172.21.15.123:0/3432763721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff06000640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b1ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 -- 172.21.15.123:0/3432763721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 msgr2=0x565091b1ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b1ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 -- 172.21.15.123:0/3432763721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 msgr2=0x565091b1e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b1e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.558+0000 7eff07400640 1 -- 172.21.15.123:0/3432763721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56509178a210 con 0x565091b1a150 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7eff07400640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x565091b15840 tx=0x7eff0000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb65c443dee28d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7efef0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:38:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00003ec0 con 0x565091b1a150 2024-03-24T00:38:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7eff00004060 con 0x565091b1a150 2024-03-24T00:38:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff00003d10 con 0x565091b1a150 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.559+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff00004260 con 0x565091b1a150 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.560+0000 7eff10bf5740 10 client.25400 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.560+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565091856620 con 0x565091b1a150 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.560+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff00003d10 con 0x565091b1a150 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7eff00031070 con 0x565091b1a150 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 10 client.25400 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 10 client.25400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 20 client.25400 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b2a3a0 0x565091b2c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 4 client.25400 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565091a82d70 con 0x565091b2a3a0 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.561+0000 7eff06a00640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b2a3a0 0x565091b2c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.562+0000 7eff06a00640 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b2a3a0 0x565091b2c790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2ded080f91db114 server_cookie=90d8f27762cb9d2a in_seq=0 out_seq=0 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.562+0000 7efef0c00640 10 client.25400 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.562+0000 7efef0c00640 1 -- 172.21.15.123:0/3432763721 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565091a82d70 con 0x565091b2a3a0 2024-03-24T00:38:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.562+0000 7efef0c00640 10 client.25400 handle_command_reply: tid=0 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.562+0000 7eff10bf5740 1 client.25400 shutdown 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 msgr2=0x565091b25d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x565091b15840 tx=0x7eff0000f730 comp rx=0 tx=0).stop 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 shutdown_connections 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b25d40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b2a3a0 0x565091b2c790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b1ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 --2- 172.21.15.123:0/3432763721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b1e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7efef0c00640 0 client.25400 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 >> 172.21.15.123:0/3432763721 conn(0x565091b0b750 msgr2=0x565091b0a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 shutdown_connections 2024-03-24T00:38:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 1 -- 172.21.15.123:0/3432763721 wait complete. 2024-03-24T00:38:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.563+0000 7eff10bf5740 20 client.25400 trim_cache size 0 max 16384 2024-03-24T00:38:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:52 smithi204 ceph-mon[36345]: pgmap v414: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.590+0000 7eff10bf5740 1 Processor -- start 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 -- start start 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b0a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b25d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a4840 con 0x565091b145f0 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a6e00 con 0x565091b1a150 2024-03-24T00:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff10bf5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a4e40 con 0x565091b14fe0 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff06a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b0a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39850/0 (socket says 172.21.15.123:39850) 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff07400640 1 -- 172.21.15.123:0/2344077870 learned_addr learned my addr 172.21.15.123:0/2344077870 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.591+0000 7eff06000640 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b25d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 -- 172.21.15.123:0/2344077870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 msgr2=0x565091b25d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b25d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 -- 172.21.15.123:0/2344077870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 msgr2=0x565091b0a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b0a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 -- 172.21.15.123:0/2344077870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565091b149d0 con 0x565091b1a150 2024-03-24T00:38:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7eff07400640 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff00005830 tx=0x7eff000092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b873210a3b32b6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7efef3400640 1 -- 172.21.15.123:0/2344077870 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff000059a0 con 0x565091b1a150 2024-03-24T00:38:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7efef3400640 1 -- 172.21.15.123:0/2344077870 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff00042020 con 0x565091b1a150 2024-03-24T00:38:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.592+0000 7efef3400640 1 -- 172.21.15.123:0/2344077870 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00005070 con 0x565091b1a150 2024-03-24T00:38:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 -- 172.21.15.123:0/2344077870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 msgr2=0x565091b0a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff00005830 tx=0x7eff000092a0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 -- 172.21.15.123:0/2344077870 shutdown_connections 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b0a600 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b25d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 --2- 172.21.15.123:0/2344077870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b0a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 -- 172.21.15.123:0/2344077870 >> 172.21.15.123:0/2344077870 conn(0x565091b0b750 msgr2=0x565091b0bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 -- 172.21.15.123:0/2344077870 shutdown_connections 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.593+0000 7eff10bf5740 1 -- 172.21.15.123:0/2344077870 wait complete. 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 -- start start 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 Processor -- start 2024-03-24T00:38:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b23a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b24a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a5480 con 0x565091b145f0 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a67a0 con 0x565091b1a150 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a5ae0 con 0x565091b14fe0 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.594+0000 7eff10bf5740 10 client.0 _ll_register_callbacks cb 0x7eff00015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53912/0 (socket says 172.21.15.123:53912) 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 -- 172.21.15.123:0/1980752925 learned_addr learned my addr 172.21.15.123:0/1980752925 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06000640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b23a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff07400640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b24a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 -- 172.21.15.123:0/1980752925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 msgr2=0x565091b23a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b23a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 -- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 msgr2=0x565091b24a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b24a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.595+0000 7eff06a00640 1 -- 172.21.15.123:0/1980752925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565091b25060 con 0x565091b145f0 2024-03-24T00:38:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7eff06a00640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x565091b15a50 tx=0x7efef800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f50d28b4938911 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7efef0c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80037f0 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efef8003990 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff00042020 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef8003b90 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7eff10bf5740 10 client.15300 fetch_fsmap learned FSMap version 19 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.596+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565091b85d70 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff00042020 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efef8026420 con 0x565091b145f0 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 10 client.15300 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 10 client.15300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 20 client.15300 populate_metadata read hostname 'smithi123' 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b86700 0x565091b88b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 4 client.15300 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565091b06a80 con 0x565091b86700 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.597+0000 7eff06000640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b86700 0x565091b88b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.598+0000 7efef0c00640 10 client.15300 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.598+0000 7eff06000640 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b86700 0x565091b88b20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7395eea722acc21 server_cookie=63792fdbc487dce in_seq=0 out_seq=0 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.601+0000 7efef0c00640 1 -- 172.21.15.123:0/1980752925 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565091b06a80 con 0x565091b86700 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.601+0000 7efef0c00640 10 client.15300 handle_command_reply: tid=0 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.601+0000 7eff10bf5740 1 client.15300 shutdown 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 msgr2=0x565091b23520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x565091b15a50 tx=0x7efef800bf00 comp rx=0 tx=0).stop 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 shutdown_connections 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565091b1a150 0x565091b24a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565091b86700 0x565091b88b20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565091b14fe0 0x565091b23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 --2- 172.21.15.123:0/1980752925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565091b145f0 0x565091b23520 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 >> 172.21.15.123:0/1980752925 conn(0x565091b0b750 msgr2=0x565091b0e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7efef0c00640 0 client.15300 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 shutdown_connections 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 1 -- 172.21.15.123:0/1980752925 wait complete. 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.602+0000 7eff10bf5740 20 client.15300 trim_cache size 0 max 16384 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:38:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 msgr2=0x7efeec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 0x7efeec0a4de0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff00011580 tx=0x7eff00009aa0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 msgr2=0x7eff08197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efef4005ab0 tx=0x7efef402c6c0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 shutdown_connections 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff08103280 0x7eff0819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081028c0 0x7eff08197e40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efeec0a2920 0x7efeec0a4de0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 --2- 172.21.15.123:0/3579033245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff081016d0 0x7eff08197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 >> 172.21.15.123:0/3579033245 conn(0x7eff080fcc60 msgr2=0x7eff08105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 shutdown_connections 2024-03-24T00:38:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:38:52.603+0000 7eff0de00640 1 -- 172.21.15.123:0/3579033245 wait complete. 2024-03-24T00:38:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:52 smithi123 ceph-mon[31487]: pgmap v414: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:52 smithi177 ceph-mon[36034]: pgmap v414: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-24T00:38:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:54 smithi123 ceph-mon[31487]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:38:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:54 smithi177 ceph-mon[36034]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:38:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:54 smithi204 ceph-mon[36345]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:38:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:56 smithi123 ceph-mon[31487]: pgmap v416: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T00:38:56.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:56 smithi204 ceph-mon[36345]: pgmap v416: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T00:38:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:56 smithi177 ceph-mon[36034]: pgmap v416: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T00:38:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:38:58 smithi123 ceph-mon[31487]: pgmap v417: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:38:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:38:58 smithi177 ceph-mon[36034]: pgmap v417: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:38:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:38:58 smithi204 ceph-mon[36345]: pgmap v417: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:00 smithi123 ceph-mon[31487]: pgmap v418: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:00 smithi177 ceph-mon[36034]: pgmap v418: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:00 smithi204 ceph-mon[36345]: pgmap v418: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: pgmap v419: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:02 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: pgmap v419: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:02 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: pgmap v419: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:02 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:39:02.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:39:03.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 -- 172.21.15.123:0/1351180014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec0104ad0 msgr2=0x7fdec0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 --2- 172.21.15.123:0/1351180014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec0104ad0 0x7fdec0104f50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009900 tx=0x7fdeac02bc40 comp rx=0 tx=0).stop 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 -- 172.21.15.123:0/1351180014 shutdown_connections 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 --2- 172.21.15.123:0/1351180014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0105490 0x7fdec010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 --2- 172.21.15.123:0/1351180014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec0104ad0 0x7fdec0104f50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 --2- 172.21.15.123:0/1351180014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec01038e0 0x7fdec0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.107+0000 7fdec7800640 1 -- 172.21.15.123:0/1351180014 >> 172.21.15.123:0/1351180014 conn(0x7fdec00fee50 msgr2=0x7fdec0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.108+0000 7fdec7800640 1 -- 172.21.15.123:0/1351180014 shutdown_connections 2024-03-24T00:39:03.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.108+0000 7fdec7800640 1 -- 172.21.15.123:0/1351180014 wait complete. 2024-03-24T00:39:03.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 Processor -- start 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 -- start start 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 0x7fdec019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec0105490 0x7fdec01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec010f590 con 0x7fdec0105490 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec010f410 con 0x7fdec01038e0 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.109+0000 7fdec7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec010f710 con 0x7fdec0104ad0 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45766/0 (socket says 172.21.15.123:45766) 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 -- 172.21.15.123:0/2814811762 learned_addr learned my addr 172.21.15.123:0/2814811762 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdebfe00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 0x7fdec019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec5800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec0105490 0x7fdec01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 -- 172.21.15.123:0/2814811762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 msgr2=0x7fdec019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 0x7fdec019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 -- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec0105490 msgr2=0x7fdec01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec0105490 0x7fdec01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdec4e00640 1 -- 172.21.15.123:0/2814811762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec01a17f0 con 0x7fdec01038e0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.110+0000 7fdebfe00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 0x7fdec019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e9c0 tx=0x7fdeb400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdebd600640 1 -- 172.21.15.123:0/2814811762 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb400cd80 con 0x7fdec01038e0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdebd600640 1 -- 172.21.15.123:0/2814811762 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeb40035f0 con 0x7fdec01038e0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdebd600640 1 -- 172.21.15.123:0/2814811762 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4010690 con 0x7fdec01038e0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec01a1ae0 con 0x7fdec01038e0 2024-03-24T00:39:03.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.111+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec01a1fb0 con 0x7fdec01038e0 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 Processor -- start 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 -- start start 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aeb6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97ae0b250 con 0x55a97aeaf080 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97adb3a30 con 0x55a97aeb41f0 2024-03-24T00:39:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.115+0000 7fdeca617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97ae072f0 con 0x55a97aeae690 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdebd600640 1 -- 172.21.15.123:0/2814811762 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdeb40037a0 con 0x7fdec01038e0 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:48888/0 (socket says 172.21.15.123:48888) 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 -- 172.21.15.123:0/2443095302 learned_addr learned my addr 172.21.15.123:0/2443095302 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdebfe00640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aeb6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 -- 172.21.15.123:0/2443095302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 msgr2=0x55a97aeb6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aeb6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 -- 172.21.15.123:0/2443095302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 msgr2=0x55a97aeb3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdec5800640 1 -- 172.21.15.123:0/2443095302 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a97ad9e690 con 0x55a97aeae690 2024-03-24T00:39:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.116+0000 7fdebfe00640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aeb6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdec5800640 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009fc0 tx=0x7fdeb8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec32488a01ab5a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdeb2000640 1 -- 172.21.15.123:0/2443095302 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8003b70 con 0x55a97aeae690 2024-03-24T00:39:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdebd600640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 0x7fdea80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdeb2000640 1 -- 172.21.15.123:0/2443095302 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeb8003d10 con 0x55a97aeae690 2024-03-24T00:39:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdeb2000640 1 -- 172.21.15.123:0/2443095302 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8011b40 con 0x55a97aeae690 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.117+0000 7fdebd600640 1 -- 172.21.15.123:0/2814811762 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdeb40ceef0 con 0x7fdec01038e0 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 0x7fdea80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 -- 172.21.15.123:0/2443095302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 msgr2=0x55a97aeaeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009fc0 tx=0x7fdeb8010860 comp rx=0 tx=0).stop 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 -- 172.21.15.123:0/2443095302 shutdown_connections 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aeb6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 --2- 172.21.15.123:0/2443095302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeaeab0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.118+0000 7fdeca617740 1 -- 172.21.15.123:0/2443095302 >> 172.21.15.123:0/2443095302 conn(0x55a97ae6e200 msgr2=0x55a97aeaa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.119+0000 7fdec4e00640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 0x7fdea80a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdeb40ccd70 tx=0x7fdeb4009790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.119+0000 7fdeca617740 1 -- 172.21.15.123:0/2443095302 shutdown_connections 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.119+0000 7fdeca617740 1 -- 172.21.15.123:0/2443095302 wait complete. 2024-03-24T00:39:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.119+0000 7fdeca617740 1 -- start start 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.119+0000 7fdeca617740 1 Processor -- start 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeb8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97aeb71a0 con 0x55a97aeaf080 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97adc6e80 con 0x55a97aeb41f0 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a97aeb7320 con 0x55a97aeae690 2024-03-24T00:39:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdeca617740 10 client.0 _ll_register_callbacks cb 0x55a97add9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:03.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdebfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdebfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33752/0 (socket says 172.21.15.123:33752) 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.120+0000 7fdebfe00640 1 -- 172.21.15.123:0/1045399714 learned_addr learned my addr 172.21.15.123:0/1045399714 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.121+0000 7fdebfe00640 1 -- 172.21.15.123:0/1045399714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 msgr2=0x55a97aeb8630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdebfe00640 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeb8630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdebfe00640 1 -- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 msgr2=0x55a97aebfbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdebfe00640 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebfbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdebfe00640 1 -- 172.21.15.123:0/1045399714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a97abe97d0 con 0x55a97aeaf080 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdebfe00640 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdeac006010 tx=0x7fdeac0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa1141991dc9365 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.122+0000 7fdea6a00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.123+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac042070 con 0x55a97aeaf080 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.123+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdeac005230 con 0x55a97aeaf080 2024-03-24T00:39:03.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.123+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdeb8003d10 con 0x55a97aeaf080 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.124+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdeac005d50 con 0x55a97aeaf080 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.124+0000 7fdeca617740 10 client.15309 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.124+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a97ab1f3f0 con 0x55a97aeaf080 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.124+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeb8003d10 con 0x55a97aeaf080 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdeac0037b0 con 0x55a97aeaf080 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 10 client.15309 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 10 client.15309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 20 client.15309 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97aec43e0 0x55a97aec6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 4 client.15309 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a97ae18790 con 0x55a97aec43e0 2024-03-24T00:39:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.125+0000 7fdec5800640 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97aec43e0 0x55a97aec6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.126+0000 7fdea6a00640 10 client.15309 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.126+0000 7fdec5800640 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97aec43e0 0x55a97aec6840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62b561c8d9798af3 server_cookie=f422db5e68589160 in_seq=0 out_seq=0 2024-03-24T00:39:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdea6a00640 1 -- 172.21.15.123:0/1045399714 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a97ae18790 con 0x55a97aec43e0 2024-03-24T00:39:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdea6a00640 10 client.15309 handle_command_reply: tid=0 2024-03-24T00:39:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 client.15309 shutdown 2024-03-24T00:39:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 msgr2=0x55a97aeb8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdeac006010 tx=0x7fdeac0045b0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 shutdown_connections 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeaf080 0x55a97aeb8b70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97aec43e0 0x55a97aec6840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdea6a00640 0 client.15309 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 --2- 172.21.15.123:0/1045399714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeae690 0x55a97aeb8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 >> 172.21.15.123:0/1045399714 conn(0x55a97ae6e200 msgr2=0x55a97aea0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 shutdown_connections 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.127+0000 7fdeca617740 1 -- 172.21.15.123:0/1045399714 wait complete. 2024-03-24T00:39:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.128+0000 7fdeca617740 20 client.15309 trim_cache size 0 max 16384 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 Processor -- start 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 -- start start 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeb41f0 0x55a97aead780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a4950 con 0x55a97aeb41f0 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a6f10 con 0x55a97aeae690 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdeca617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a4f50 con 0x55a97aeaf080 2024-03-24T00:39:03.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdec5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdec5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45792/0 (socket says 172.21.15.123:45792) 2024-03-24T00:39:03.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdec5800640 1 -- 172.21.15.123:0/626162062 learned_addr learned my addr 172.21.15.123:0/626162062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdec5800640 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeb41f0 0x55a97aead780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.146+0000 7fdebfe00640 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 -- 172.21.15.123:0/626162062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 msgr2=0x55a97aebfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 -- 172.21.15.123:0/626162062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeb41f0 msgr2=0x55a97aead780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeb41f0 0x55a97aead780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 -- 172.21.15.123:0/626162062 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a97aadf0f0 con 0x55a97aeae690 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdec5800640 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009f60 tx=0x7fdeb8009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b2b439e9e1c56e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdeb1600640 1 -- 172.21.15.123:0/626162062 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8011aa0 con 0x55a97aeae690 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdeb1600640 1 -- 172.21.15.123:0/626162062 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeb8011c40 con 0x55a97aeae690 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.147+0000 7fdeb1600640 1 -- 172.21.15.123:0/626162062 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8026410 con 0x55a97aeae690 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 -- 172.21.15.123:0/626162062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 msgr2=0x55a97aead240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009f60 tx=0x7fdeb8009bd0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 -- 172.21.15.123:0/626162062 shutdown_connections 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeb41f0 0x55a97aead780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 --2- 172.21.15.123:0/626162062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeae690 0x55a97aead240 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 -- 172.21.15.123:0/626162062 >> 172.21.15.123:0/626162062 conn(0x55a97ae6e200 msgr2=0x55a97aeb03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 -- 172.21.15.123:0/626162062 shutdown_connections 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.148+0000 7fdeca617740 1 -- 172.21.15.123:0/626162062 wait complete. 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 -- start start 2024-03-24T00:39:03.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 Processor -- start 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeae690 0x55a97aebd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a5590 con 0x55a97aeae690 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a68b0 con 0x55a97aeb41f0 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a5bf0 con 0x55a97aeaf080 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdeca617740 10 client.0 _ll_register_callbacks cb 0x7fdeb4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:03.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.149+0000 7fdec5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeae690 0x55a97aebd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdec5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdec5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45794/0 (socket says 172.21.15.123:45794) 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdec5800640 1 -- 172.21.15.123:0/150719944 learned_addr learned my addr 172.21.15.123:0/150719944 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 -- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 msgr2=0x55a97aebeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 -- 172.21.15.123:0/150719944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeae690 msgr2=0x55a97aebd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeae690 0x55a97aebd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 -- 172.21.15.123:0/150719944 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a97aebf150 con 0x55a97aeaf080 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdebfe00640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebdb50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeac003b80 tx=0x7fdeac003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75300cfdf68b7bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.150+0000 7fdea6a00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac042070 con 0x55a97aeaf080 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdeac005410 con 0x55a97aeaf080 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdeb8011c40 con 0x55a97aeaf080 2024-03-24T00:39:03.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdeac009520 con 0x55a97aeaf080 2024-03-24T00:39:03.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdeca617740 10 client.25414 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:03.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.151+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a97af1ff90 con 0x55a97aeaf080 2024-03-24T00:39:03.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeb8011c40 con 0x55a97aeaf080 2024-03-24T00:39:03.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdeac0346b0 con 0x55a97aeaf080 2024-03-24T00:39:03.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 10 client.25414 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 10 client.25414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 20 client.25414 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97af208b0 0x55a97af22cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 4 client.25414 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a97aeb8420 con 0x55a97af208b0 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.152+0000 7fdec5800640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97af208b0 0x55a97af22cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdec5800640 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97af208b0 0x55a97af22cd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e63ae9e3b2bac48e server_cookie=29e79fd1a3bd1557 in_seq=0 out_seq=0 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdea6a00640 10 client.25414 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdea6a00640 1 -- 172.21.15.123:0/150719944 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a97aeb8420 con 0x55a97af208b0 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdea6a00640 10 client.25414 handle_command_reply: tid=0 2024-03-24T00:39:03.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 client.25414 shutdown 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 msgr2=0x55a97aebdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebdb50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeac003b80 tx=0x7fdeac003820 comp rx=0 tx=0).stop 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 shutdown_connections 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a97aeb41f0 0x55a97aebeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55a97af208b0 0x55a97af22cd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdea6a00640 0 client.25414 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a97aeaf080 0x55a97aebdb50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 --2- 172.21.15.123:0/150719944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a97aeae690 0x55a97aebd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 >> 172.21.15.123:0/150719944 conn(0x55a97ae6e200 msgr2=0x55a97aea0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 shutdown_connections 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.153+0000 7fdeca617740 1 -- 172.21.15.123:0/150719944 wait complete. 2024-03-24T00:39:03.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdeca617740 20 client.25414 trim_cache size 0 max 16384 2024-03-24T00:39:03.158 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:39:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:39:03.158 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:03.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 msgr2=0x7fdea80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 0x7fdea80a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdeb40ccd70 tx=0x7fdeb4009790 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 msgr2=0x7fdec0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e9c0 tx=0x7fdeb400ee80 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 shutdown_connections 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdea80a2880 0x7fdea80a4d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec0105490 0x7fdec01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdec0104ad0 0x7fdec019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 --2- 172.21.15.123:0/2814811762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01038e0 0x7fdec0199bd0 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e9c0 tx=0x7fdeb400ee80 comp rx=0 tx=0).stop 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.154+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 >> 172.21.15.123:0/2814811762 conn(0x7fdec00fee50 msgr2=0x7fdec0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.155+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 shutdown_connections 2024-03-24T00:39:03.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.155+0000 7fdec7800640 1 -- 172.21.15.123:0/2814811762 wait complete. 2024-03-24T00:39:03.171 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:39:03.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 -- 172.21.15.123:0/67931533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840102140 msgr2=0x7f8840104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 --2- 172.21.15.123:0/67931533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840102140 0x7f8840104530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8830009900 tx=0x7f883002bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 -- 172.21.15.123:0/67931533 shutdown_connections 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 --2- 172.21.15.123:0/67931533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88401073a0 0x7f88401097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 --2- 172.21.15.123:0/67931533 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8840104a70 0x7f8840106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 --2- 172.21.15.123:0/67931533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840102140 0x7f8840104530 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 -- 172.21.15.123:0/67931533 >> 172.21.15.123:0/67931533 conn(0x7f88400fbb50 msgr2=0x7f88400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 -- 172.21.15.123:0/67931533 shutdown_connections 2024-03-24T00:39:03.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.347+0000 7f8847c00640 1 -- 172.21.15.123:0/67931533 wait complete. 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 Processor -- start 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 -- start start 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840104a70 0x7f8840195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88401073a0 0x7f884019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.348+0000 7f8847c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884010b190 con 0x7f8840102140 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8847c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884010b010 con 0x7f8840104a70 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8847c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884010b310 con 0x7f88401073a0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88401073a0 0x7f884019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33780/0 (socket says 172.21.15.123:33780) 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 -- 172.21.15.123:0/3674005836 learned_addr learned my addr 172.21.15.123:0/3674005836 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8837e00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840104a70 0x7f8840195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 -- 172.21.15.123:0/3674005836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88401073a0 msgr2=0x7f884019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88401073a0 0x7f884019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 -- 172.21.15.123:0/3674005836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840104a70 msgr2=0x7f8840195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840104a70 0x7f8840195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.349+0000 7f8845200640 1 -- 172.21.15.123:0/3674005836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884006b1f0 con 0x7f8840102140 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f8845200640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f883002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830043020 con 0x7f8840102140 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884006b480 con 0x7f8840102140 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88300054e0 con 0x7f8840102140 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830003bd0 con 0x7f8840102140 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.350+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884006ba20 con 0x7f8840102140 2024-03-24T00:39:03.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.352+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88400fd640 con 0x7f8840102140 2024-03-24T00:39:03.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.354+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8830046020 con 0x7f8840102140 2024-03-24T00:39:03.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.355+0000 7f883e000640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 0x7f88240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.355+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f88300bb4f0 con 0x7f8840102140 2024-03-24T00:39:03.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.355+0000 7f8837e00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 0x7f88240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.356+0000 7f8837e00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 0x7f88240a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8840196d00 tx=0x7f882800f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.361+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f883004d020 con 0x7f8840102140 2024-03-24T00:39:03.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.593+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f884002d070 con 0x7f8840102140 2024-03-24T00:39:03.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.594+0000 7f883e000640 1 -- 172.21.15.123:0/3674005836 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f88300bdb50 con 0x7f8840102140 2024-03-24T00:39:03.595 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:39:03.595 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:39:03.595 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 msgr2=0x7f88240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 0x7f88240a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8840196d00 tx=0x7f882800f040 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 msgr2=0x7f8840195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f883002c650 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 shutdown_connections 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88401073a0 0x7f884019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8840104a70 0x7f8840195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f88240a2800 0x7f88240a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 --2- 172.21.15.123:0/3674005836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8840102140 0x7f8840195840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 >> 172.21.15.123:0/3674005836 conn(0x7f88400fbb50 msgr2=0x7f8840102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 shutdown_connections 2024-03-24T00:39:03.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.597+0000 7f8847c00640 1 -- 172.21.15.123:0/3674005836 wait complete. 2024-03-24T00:39:03.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:03.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:03.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:03 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:03.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:03 smithi123 ceph-mon[31487]: pgmap v420: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:03.619 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:39:03.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:39:03.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:03.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:03.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:03 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:03.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:03 smithi177 ceph-mon[36034]: pgmap v420: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 -- 172.21.15.123:0/717342012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf40756e0 msgr2=0x7fcdf4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 --2- 172.21.15.123:0/717342012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf40756e0 0x7fcdf4073b30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009900 tx=0x7fcde802bc40 comp rx=0 tx=0).stop 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 -- 172.21.15.123:0/717342012 shutdown_connections 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 --2- 172.21.15.123:0/717342012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf4109460 0x7fcdf410b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 --2- 172.21.15.123:0/717342012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4074100 0x7fcdf4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 --2- 172.21.15.123:0/717342012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf40756e0 0x7fcdf4073b30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 -- 172.21.15.123:0/717342012 >> 172.21.15.123:0/717342012 conn(0x7fcdf40fcb10 msgr2=0x7fcdf40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 -- 172.21.15.123:0/717342012 shutdown_connections 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.791+0000 7fcdfae00640 1 -- 172.21.15.123:0/717342012 wait complete. 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.792+0000 7fcdfae00640 1 Processor -- start 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.792+0000 7fcdfae00640 1 -- start start 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf40756e0 0x7fcdf4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4109460 0x7fcdf41a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410f430 con 0x7fcdf40756e0 2024-03-24T00:39:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410f2b0 con 0x7fcdf4074100 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410f5b0 con 0x7fcdf4109460 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45836/0 (socket says 172.21.15.123:45836) 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 -- 172.21.15.123:0/3939510279 learned_addr learned my addr 172.21.15.123:0/3939510279 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9000640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf40756e0 0x7fcdf4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdfa400640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4109460 0x7fcdf41a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 -- 172.21.15.123:0/3939510279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4109460 msgr2=0x7fcdf41a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4109460 0x7fcdf41a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 -- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf40756e0 msgr2=0x7fcdf4199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf40756e0 0x7fcdf4199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.793+0000 7fcdf9a00640 1 -- 172.21.15.123:0/3939510279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf41a16a0 con 0x7fcdf4074100 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcde802c8f0 tx=0x7fcde802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdf2000640 1 -- 172.21.15.123:0/3939510279 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8041070 con 0x7fcdf4074100 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdf2000640 1 -- 172.21.15.123:0/3939510279 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde8004450 con 0x7fcdf4074100 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdf2000640 1 -- 172.21.15.123:0/3939510279 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8034440 con 0x7fcdf4074100 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf41a1930 con 0x7fcdf4074100 2024-03-24T00:39:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.794+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf41a1dd0 con 0x7fcdf4074100 2024-03-24T00:39:03.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.798+0000 7fcdff19e740 1 Processor -- start 2024-03-24T00:39:03.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 -- start start 2024-03-24T00:39:03.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cf9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 0x5577b6cfb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6c5e2c0 con 0x5577b6cfabc0 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6c06c90 con 0x5577b6cfb580 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdff19e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6c5a360 con 0x5577b6cf99f0 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf2000640 1 -- 172.21.15.123:0/3939510279 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcde8044070 con 0x7fcdf4074100 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 0x5577b6cfb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45848/0 (socket says 172.21.15.123:45848) 2024-03-24T00:39:03.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf9000640 1 -- 172.21.15.123:0/1084251552 learned_addr learned my addr 172.21.15.123:0/1084251552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdfa400640 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cf9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.799+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 0x5577b6cfb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33804/0 (socket says 172.21.15.123:33804) 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 -- 172.21.15.123:0/1084251552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 msgr2=0x5577b6cf9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cf9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 -- 172.21.15.123:0/1084251552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 msgr2=0x5577b6cfb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 0x5577b6cfb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 -- 172.21.15.123:0/1084251552 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577b6c2b080 con 0x5577b6cfb580 2024-03-24T00:39:03.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.800+0000 7fcdf9000640 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009c90 tx=0x7fcde0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf9c87e5dd083a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcde6a00640 1 -- 172.21.15.123:0/1084251552 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde001e070 con 0x5577b6cfb580 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcde6a00640 1 -- 172.21.15.123:0/1084251552 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde0019b10 con 0x5577b6cfb580 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcde6a00640 1 -- 172.21.15.123:0/1084251552 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000eac0 con 0x5577b6cfb580 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 -- 172.21.15.123:0/1084251552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 msgr2=0x5577b6d01e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009c90 tx=0x7fcde0011810 comp rx=0 tx=0).stop 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 -- 172.21.15.123:0/1084251552 shutdown_connections 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d01e10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfabc0 0x5577b6cfb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 --2- 172.21.15.123:0/1084251552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cf9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdff19e740 1 -- 172.21.15.123:0/1084251552 >> 172.21.15.123:0/1084251552 conn(0x5577b6cf4de0 msgr2=0x5577b6cf51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.801+0000 7fcdf2000640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 0x7fcddc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.802+0000 7fcdff19e740 1 -- 172.21.15.123:0/1084251552 shutdown_connections 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.802+0000 7fcdf2000640 1 -- 172.21.15.123:0/3939510279 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcde80f05f0 con 0x7fcdf4074100 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.802+0000 7fcdfa400640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 0x7fcddc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.802+0000 7fcdff19e740 1 -- 172.21.15.123:0/1084251552 wait complete. 2024-03-24T00:39:03.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.803+0000 7fcdff19e740 1 -- start start 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.803+0000 7fcdff19e740 1 Processor -- start 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.803+0000 7fcdfa400640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 0x7fcddc0a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5577b6cfaac0 tx=0x7fcdec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.803+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 0x5577b6d03c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cfabc0 0x5577b6d041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6d02590 con 0x5577b6cf99f0 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6c1a0d0 con 0x5577b6cfb580 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577b6d02710 con 0x5577b6cfabc0 2024-03-24T00:39:03.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdff19e740 10 client.0 _ll_register_callbacks cb 0x5577b6c1ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:03.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdfa400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdfa400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45862/0 (socket says 172.21.15.123:45862) 2024-03-24T00:39:03.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdfa400640 1 -- 172.21.15.123:0/2347033968 learned_addr learned my addr 172.21.15.123:0/2347033968 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdf9000640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cfabc0 0x5577b6d041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.804+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 0x5577b6d03c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 -- 172.21.15.123:0/2347033968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cfabc0 msgr2=0x5577b6d041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cfabc0 0x5577b6d041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 -- 172.21.15.123:0/2347033968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 msgr2=0x5577b6d03c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 0x5577b6d03c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 -- 172.21.15.123:0/2347033968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577b69723f0 con 0x5577b6cfb580 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 0x5577b6d03c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcdfa400640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009f50 tx=0x7fcdec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4161966bd6852d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.805+0000 7fcddb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00f7b0 con 0x5577b6cfb580 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcde0019b10 con 0x5577b6cfb580 2024-03-24T00:39:03.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcdec00f950 con 0x5577b6cfb580 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdec00fb50 con 0x5577b6cfb580 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcdff19e740 10 client.25439 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5577b6a3c7d0 con 0x5577b6cfb580 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.806+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde0019b10 con 0x5577b6cfb580 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcdec026070 con 0x5577b6cfb580 2024-03-24T00:39:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 10 client.25439 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 10 client.25439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 20 client.25439 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d17ee0 0x5577b6d1a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 4 client.25439 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577b6c6b800 con 0x5577b6d17ee0 2024-03-24T00:39:03.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d17ee0 0x5577b6d1a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcddb400640 10 client.25439 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.807+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d17ee0 0x5577b6d1a350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a36791f43ba2c155 server_cookie=262ac9e86db10128 in_seq=0 out_seq=0 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcddb400640 1 -- 172.21.15.123:0/2347033968 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5577b6c6b800 con 0x5577b6d17ee0 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcddb400640 10 client.25439 handle_command_reply: tid=0 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 client.25439 shutdown 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 msgr2=0x5577b6d0b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009f50 tx=0x7fcdec004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 shutdown_connections 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfb580 0x5577b6d0b1e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d17ee0 0x5577b6d1a350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cfabc0 0x5577b6d041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 --2- 172.21.15.123:0/2347033968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cf99f0 0x5577b6d03c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 >> 172.21.15.123:0/2347033968 conn(0x5577b6cf4de0 msgr2=0x5577b6cf4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.808+0000 7fcddb400640 0 client.25439 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.809+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 shutdown_connections 2024-03-24T00:39:03.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.809+0000 7fcdff19e740 1 -- 172.21.15.123:0/2347033968 wait complete. 2024-03-24T00:39:03.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.809+0000 7fcdff19e740 20 client.25439 trim_cache size 0 max 16384 2024-03-24T00:39:03.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:03.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:03.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:03 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:39:03.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:03 smithi204 ceph-mon[36345]: pgmap v420: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:03.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 Processor -- start 2024-03-24T00:39:03.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 -- start start 2024-03-24T00:39:03.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6cfe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a46c0 con 0x5577b6cfb580 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a6c80 con 0x5577b6cfabc0 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.822+0000 7fcdff19e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a4cc0 con 0x5577b6cf99f0 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:48982/0 (socket says 172.21.15.123:48982) 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 -- 172.21.15.123:0/80045498 learned_addr learned my addr 172.21.15.123:0/80045498 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdfa400640 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6cfe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 -- 172.21.15.123:0/80045498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 msgr2=0x5577b6d0b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 -- 172.21.15.123:0/80045498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 msgr2=0x5577b6cfe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6cfe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 -- 172.21.15.123:0/80045498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577b6cf08b0 con 0x5577b6cf99f0 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdfa400640 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6cfe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcde8006010 tx=0x7fcde80edf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacea1ce5c8ad1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcde6000640 1 -- 172.21.15.123:0/80045498 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde809be60 con 0x5577b6cf99f0 2024-03-24T00:39:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.823+0000 7fcde6000640 1 -- 172.21.15.123:0/80045498 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde80bcff0 con 0x5577b6cf99f0 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcde6000640 1 -- 172.21.15.123:0/80045498 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde804b040 con 0x5577b6cf99f0 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 -- 172.21.15.123:0/80045498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 msgr2=0x5577b6cfddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcde8006010 tx=0x7fcde80edf70 comp rx=0 tx=0).stop 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 -- 172.21.15.123:0/80045498 shutdown_connections 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6cfe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 --2- 172.21.15.123:0/80045498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6cfddf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 -- 172.21.15.123:0/80045498 >> 172.21.15.123:0/80045498 conn(0x5577b6cf4de0 msgr2=0x5577b6cf88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 -- 172.21.15.123:0/80045498 shutdown_connections 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.824+0000 7fcdff19e740 1 -- 172.21.15.123:0/80045498 wait complete. 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 -- start start 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 Processor -- start 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6d0aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6d09160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a5300 con 0x5577b6cfb580 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a6620 con 0x5577b6cfabc0 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a5960 con 0x5577b6cf99f0 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.825+0000 7fcdff19e740 10 client.0 _ll_register_callbacks cb 0x7fcdec009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45888/0 (socket says 172.21.15.123:45888) 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 -- 172.21.15.123:0/3711017923 learned_addr learned my addr 172.21.15.123:0/3711017923 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6d0aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdfa400640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6d09160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 -- 172.21.15.123:0/3711017923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 msgr2=0x5577b6d0aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6d0aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 -- 172.21.15.123:0/3711017923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 msgr2=0x5577b6d09160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6d09160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 -- 172.21.15.123:0/3711017923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577b6d09990 con 0x5577b6cfabc0 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdfa400640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6d09160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcdf9000640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009c90 tx=0x7fcde0005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af6930aa885211e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.826+0000 7fcddb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde001e040 con 0x5577b6cfabc0 2024-03-24T00:39:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcde80bcff0 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcde00055f0 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcde0005c50 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcdff19e740 10 client.25445 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5577b6d74300 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde80bcff0 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.827+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcde002b020 con 0x5577b6cfabc0 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 10 client.25445 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 10 client.25445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 20 client.25445 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d74a20 0x5577b6d76e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 4 client.25445 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5577b6c65b50 con 0x5577b6d74a20 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d74a20 0x5577b6d76e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcdf9a00640 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d74a20 0x5577b6d76e40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9997f6c5295305 server_cookie=1caf65130b6c4f91 in_seq=0 out_seq=0 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcddb400640 10 client.25445 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcddb400640 1 -- 172.21.15.123:0/3711017923 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5577b6c65b50 con 0x5577b6d74a20 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.828+0000 7fcddb400640 10 client.25445 handle_command_reply: tid=0 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 client.25445 shutdown 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 msgr2=0x5577b6d0b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009c90 tx=0x7fcde0005270 comp rx=0 tx=0).stop 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 shutdown_connections 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5577b6cfb580 0x5577b6d09160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577b6cfabc0 0x5577b6d0b020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5577b6d74a20 0x5577b6d76e40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 --2- 172.21.15.123:0/3711017923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5577b6cf99f0 0x5577b6d0aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcddb400640 0 client.25445 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 >> 172.21.15.123:0/3711017923 conn(0x5577b6cf4de0 msgr2=0x5577b6cf38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 shutdown_connections 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 1 -- 172.21.15.123:0/3711017923 wait complete. 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.829+0000 7fcdff19e740 20 client.25445 trim_cache size 0 max 16384 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:39:03.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 msgr2=0x7fcddc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 0x7fcddc0a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5577b6cfaac0 tx=0x7fcdec009500 comp rx=0 tx=0).stop 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 msgr2=0x7fcdf4199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcde802c8f0 tx=0x7fcde802c660 comp rx=0 tx=0).stop 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 shutdown_connections 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdf4109460 0x7fcdf41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcddc0a2880 0x7fcddc0a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf40756e0 0x7fcdf4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 --2- 172.21.15.123:0/3939510279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf4074100 0x7fcdf4199a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 >> 172.21.15.123:0/3939510279 conn(0x7fcdf40fcb10 msgr2=0x7fcdf4105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 shutdown_connections 2024-03-24T00:39:03.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:03.830+0000 7fcdfae00640 1 -- 172.21.15.123:0/3939510279 wait complete. 2024-03-24T00:39:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:04 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3674005836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:04 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3674005836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:04 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3674005836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:04.851 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:04.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 -- 172.21.15.123:0/447048335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 msgr2=0x7f46701099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 --2- 172.21.15.123:0/447048335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f46701099e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4668009ca0 tx=0x7f466802c100 comp rx=0 tx=0).stop 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 -- 172.21.15.123:0/447048335 shutdown_connections 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 --2- 172.21.15.123:0/447048335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f46701099e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 --2- 172.21.15.123:0/447048335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 --2- 172.21.15.123:0/447048335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 0x7f46701019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 -- 172.21.15.123:0/447048335 >> 172.21.15.123:0/447048335 conn(0x7f46700fcb10 msgr2=0x7f46700fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 -- 172.21.15.123:0/447048335 shutdown_connections 2024-03-24T00:39:05.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.133+0000 7f4676600640 1 -- 172.21.15.123:0/447048335 wait complete. 2024-03-24T00:39:05.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 Processor -- start 2024-03-24T00:39:05.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 -- start start 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 0x7f4670197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010d230 con 0x7f4670102790 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010d0b0 con 0x7f4670103150 2024-03-24T00:39:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.135+0000 7f4676600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010d3b0 con 0x7f46701015a0 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45912/0 (socket says 172.21.15.123:45912) 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 -- 172.21.15.123:0/1488673262 learned_addr learned my addr 172.21.15.123:0/1488673262 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f466fe00640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675200640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 0x7f4670197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 -- 172.21.15.123:0/1488673262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 msgr2=0x7f4670197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 0x7f4670197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 -- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 msgr2=0x7f4670197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f4675c00640 1 -- 172.21.15.123:0/1488673262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467019f440 con 0x7f4670103150 2024-03-24T00:39:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.136+0000 7f466fe00640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f4675c00640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f466802c640 tx=0x7f4668035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f466d600640 1 -- 172.21.15.123:0/1488673262 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668039040 con 0x7f4670103150 2024-03-24T00:39:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f466d600640 1 -- 172.21.15.123:0/1488673262 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4668044070 con 0x7f4670103150 2024-03-24T00:39:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467019f6d0 con 0x7f4670103150 2024-03-24T00:39:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f466d600640 1 -- 172.21.15.123:0/1488673262 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668004460 con 0x7f4670103150 2024-03-24T00:39:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.137+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467019fc10 con 0x7f4670103150 2024-03-24T00:39:05.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.141+0000 7f467a858740 1 Processor -- start 2024-03-24T00:39:05.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.143+0000 7f466d600640 1 -- 172.21.15.123:0/1488673262 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4668004600 con 0x7f4670103150 2024-03-24T00:39:05.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.143+0000 7f467a858740 1 -- start start 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926820b0 0x561f926824d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 0x561f92687030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f925deeb0 con 0x561f92682aa0 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f92587a60 con 0x561f926876a0 2024-03-24T00:39:05.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f467a858740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f925daf50 con 0x561f926820b0 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.144+0000 7f466fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45914/0 (socket says 172.21.15.123:45914) 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f4675200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 0x561f92687030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 -- 172.21.15.123:0/3288117949 learned_addr learned my addr 172.21.15.123:0/3288117949 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926820b0 0x561f926824d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 -- 172.21.15.123:0/3288117949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926820b0 msgr2=0x561f926824d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926820b0 0x561f926824d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 -- 172.21.15.123:0/3288117949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 msgr2=0x561f92687030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 0x561f92687030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f466fe00640 1 -- 172.21.15.123:0/3288117949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f923bb7e0 con 0x561f926876a0 2024-03-24T00:39:05.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.145+0000 7f4675200640 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 0x561f92687030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f466d600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 0x7f46580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f466fe00640 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f465c0127f0 tx=0x7f465c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6db1803b09eefad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f466d600640 1 -- 172.21.15.123:0/1488673262 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f46680f0ec0 con 0x7f4670103150 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f4662000640 1 -- 172.21.15.123:0/3288117949 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c012e30 con 0x561f926876a0 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f4662000640 1 -- 172.21.15.123:0/3288117949 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f465c0107d0 con 0x561f926876a0 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f4662000640 1 -- 172.21.15.123:0/3288117949 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c003850 con 0x561f926876a0 2024-03-24T00:39:05.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.146+0000 7f4675200640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 0x7f46580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 -- 172.21.15.123:0/3288117949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 msgr2=0x561f92689b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f465c0127f0 tx=0x7f465c009720 comp rx=0 tx=0).stop 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 -- 172.21.15.123:0/3288117949 shutdown_connections 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f926876a0 0x561f92689b10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f92682aa0 0x561f92687030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 --2- 172.21.15.123:0/3288117949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926820b0 0x561f926824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.147+0000 7f467a858740 1 -- 172.21.15.123:0/3288117949 >> 172.21.15.123:0/3288117949 conn(0x561f926747a0 msgr2=0x561f92674ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.148+0000 7f4675200640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 0x7f46580a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x561f92682f40 tx=0x7f466400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.148+0000 7f467a858740 1 -- 172.21.15.123:0/3288117949 shutdown_connections 2024-03-24T00:39:05.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.148+0000 7f467a858740 1 -- 172.21.15.123:0/3288117949 wait complete. 2024-03-24T00:39:05.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.149+0000 7f467a858740 1 -- start start 2024-03-24T00:39:05.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.149+0000 7f467a858740 1 Processor -- start 2024-03-24T00:39:05.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f9268c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92693370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f9268a630 con 0x561f926820b0 2024-03-24T00:39:05.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f9259aeb0 con 0x561f92682aa0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f9268a7b0 con 0x561f926876a0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f467a858740 10 client.0 _ll_register_callbacks cb 0x561f9258f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f4675200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f4675c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92693370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f4675200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33882/0 (socket says 172.21.15.123:33882) 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f4675200640 1 -- 172.21.15.123:0/530418894 learned_addr learned my addr 172.21.15.123:0/530418894 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f466fe00640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f9268c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f466fe00640 1 -- 172.21.15.123:0/530418894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 msgr2=0x561f92693370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f466fe00640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92693370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.150+0000 7f466fe00640 1 -- 172.21.15.123:0/530418894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 msgr2=0x561f9268bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.151+0000 7f466fe00640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.151+0000 7f466fe00640 1 -- 172.21.15.123:0/530418894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f9241bf70 con 0x561f92682aa0 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.151+0000 7f4675200640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.151+0000 7f466fe00640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f9268c330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f465c0107a0 tx=0x7f465c010bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed5642e23f2cc0e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.151+0000 7f4656a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c005280 con 0x561f92682aa0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f465c005420 con 0x561f92682aa0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f465c0107d0 con 0x561f92682aa0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f465c02f050 con 0x561f92682aa0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f467a858740 10 client.25457 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.152+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561f922b3170 con 0x561f92682aa0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.153+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f465c0107d0 con 0x561f92682aa0 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f465c003710 con 0x561f92682aa0 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 10 client.25457 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 10 client.25457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 20 client.25457 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f92697860 0x561f92699c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 4 client.25457 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f925ec3f0 con 0x561f92697860 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.154+0000 7f4675200640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f92697860 0x561f92699c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.155+0000 7f4675200640 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f92697860 0x561f92699c50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8da9963dbae94321 server_cookie=88561659765499c3 in_seq=0 out_seq=0 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.155+0000 7f4656a00640 10 client.25457 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.156+0000 7f4656a00640 1 -- 172.21.15.123:0/530418894 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561f925ec3f0 con 0x561f92697860 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.156+0000 7f4656a00640 10 client.25457 handle_command_reply: tid=0 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.156+0000 7f467a858740 1 client.25457 shutdown 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.156+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 msgr2=0x561f9268c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.156+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f9268c330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f465c0107a0 tx=0x7f465c010bb0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 shutdown_connections 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f92697860 0x561f92699c50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92693370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f9268c330 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 --2- 172.21.15.123:0/530418894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f9268bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 >> 172.21.15.123:0/530418894 conn(0x561f926747a0 msgr2=0x561f92681740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.157+0000 7f4656a00640 0 client.25457 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.158+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 shutdown_connections 2024-03-24T00:39:05.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.158+0000 7f467a858740 1 -- 172.21.15.123:0/530418894 wait complete. 2024-03-24T00:39:05.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.158+0000 7f467a858740 20 client.25457 trim_cache size 0 max 16384 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.191+0000 7f467a858740 1 Processor -- start 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.191+0000 7f467a858740 1 -- start start 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.191+0000 7f467a858740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.191+0000 7f467a858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f467a858740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92682fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f467a858740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a4730 con 0x561f926820b0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f467a858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a6cf0 con 0x561f92682aa0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f467a858740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a4d30 con 0x561f926876a0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f4675c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92682fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45932/0 (socket says 172.21.15.123:45932) 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 -- 172.21.15.123:0/3839684736 learned_addr learned my addr 172.21.15.123:0/3839684736 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f4675200640 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 -- 172.21.15.123:0/3839684736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 msgr2=0x561f92682fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92682fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 -- 172.21.15.123:0/3839684736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 msgr2=0x561f926815c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926815c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f466fe00640 1 -- 172.21.15.123:0/3839684736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f92686b30 con 0x561f92682aa0 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.192+0000 7f4675c00640 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92682fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.193+0000 7f466fe00640 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f465c022040 tx=0x7f465c009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8324a8fc2b141ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.193+0000 7f4661600640 1 -- 172.21.15.123:0/3839684736 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0107d0 con 0x561f92682aa0 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.193+0000 7f4661600640 1 -- 172.21.15.123:0/3839684736 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f465c004450 con 0x561f92682aa0 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.193+0000 7f4661600640 1 -- 172.21.15.123:0/3839684736 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0035f0 con 0x561f92682aa0 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 -- 172.21.15.123:0/3839684736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 msgr2=0x561f92693330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f465c022040 tx=0x7f465c009970 comp rx=0 tx=0).stop 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 -- 172.21.15.123:0/3839684736 shutdown_connections 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92682fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92693330 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 --2- 172.21.15.123:0/3839684736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 -- 172.21.15.123:0/3839684736 >> 172.21.15.123:0/3839684736 conn(0x561f926747a0 msgr2=0x561f92680eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 -- 172.21.15.123:0/3839684736 shutdown_connections 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.194+0000 7f467a858740 1 -- 172.21.15.123:0/3839684736 wait complete. 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.195+0000 7f467a858740 1 -- start start 2024-03-24T00:39:05.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.195+0000 7f467a858740 1 Processor -- start 2024-03-24T00:39:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.195+0000 7f467a858740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.195+0000 7f467a858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92691610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.195+0000 7f467a858740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f467a858740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a5370 con 0x561f926820b0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f467a858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a6690 con 0x561f92682aa0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f467a858740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701a59d0 con 0x561f926876a0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f467a858740 10 client.0 _ll_register_callbacks cb 0x7f466400b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49032/0 (socket says 172.21.15.123:49032) 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 -- 172.21.15.123:0/2159009690 learned_addr learned my addr 172.21.15.123:0/2159009690 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f466fe00640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92691610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 -- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 msgr2=0x561f92691610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92691610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 -- 172.21.15.123:0/2159009690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 msgr2=0x561f926910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.196+0000 7f4675c00640 1 -- 172.21.15.123:0/2159009690 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f92692c10 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f4675c00640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4668032bc0 tx=0x7f466802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2081ed4d61fda3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f4656a00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680bd870 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4668032890 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f465c004450 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46680c19c0 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f467a858740 10 client.25456 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.197+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561f926f37c0 con 0x561f926876a0 2024-03-24T00:39:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46680c6010 con 0x561f926876a0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f466803c030 con 0x561f926876a0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 10 client.25456 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 20 client.25456 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f926f40f0 0x561f926f6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 4 client.25456 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7f465c004450 con 0x561f926f40f0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.198+0000 7f4675200640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f926f40f0 0x561f926f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.199+0000 7f4675200640 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f926f40f0 0x561f926f6510 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62dbcae50c5d82d8 server_cookie=eb7ea4c2e0ef2d10 in_seq=0 out_seq=0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.199+0000 7f4656a00640 10 client.25456 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f4656a00640 1 -- 172.21.15.123:0/2159009690 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f465c004450 con 0x561f926f40f0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f4656a00640 10 client.25456 handle_command_reply: tid=0 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 client.25456 shutdown 2024-03-24T00:39:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 msgr2=0x561f92692620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4668032bc0 tx=0x7f466802e040 comp rx=0 tx=0).stop 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 shutdown_connections 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f926f40f0 0x561f926f6510 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f926876a0 0x561f92692620 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f92682aa0 0x561f92691610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 --2- 172.21.15.123:0/2159009690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f926820b0 0x561f926910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f4656a00640 0 client.25456 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.200+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 >> 172.21.15.123:0/2159009690 conn(0x561f926747a0 msgr2=0x561f92681740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.201+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 shutdown_connections 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.201+0000 7f467a858740 1 -- 172.21.15.123:0/2159009690 wait complete. 2024-03-24T00:39:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.201+0000 7f467a858740 20 client.25456 trim_cache size 0 max 16384 2024-03-24T00:39:05.206 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:05.206 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:39:05.206 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "77394ba9-ece8-41c2-b3c4-da5f5090ad13", 2024-03-24T00:39:05.206 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:39:05.206 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 msgr2=0x7f46580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 0x7f46580a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x561f92682f40 tx=0x7f466400b390 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 msgr2=0x7f467019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f466802c640 tx=0x7f4668035d10 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 shutdown_connections 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4670103150 0x7f467019edd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f46580a2880 0x7f46580a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4670102790 0x7f4670197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 --2- 172.21.15.123:0/1488673262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46701015a0 0x7f4670197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 >> 172.21.15.123:0/1488673262 conn(0x7f46700fcb10 msgr2=0x7f4670105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 shutdown_connections 2024-03-24T00:39:05.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.202+0000 7f4676600640 1 -- 172.21.15.123:0/1488673262 wait complete. 2024-03-24T00:39:05.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:39:05.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:39:05.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 -- 172.21.15.123:0/95423165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc104930 msgr2=0x7ffafc106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 --2- 172.21.15.123:0/95423165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc104930 0x7ffafc106d20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009900 tx=0x7ffae802bc40 comp rx=0 tx=0).stop 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 -- 172.21.15.123:0/95423165 shutdown_connections 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 --2- 172.21.15.123:0/95423165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc107260 0x7ffafc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 --2- 172.21.15.123:0/95423165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc104930 0x7ffafc106d20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 --2- 172.21.15.123:0/95423165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 0x7ffafc1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 -- 172.21.15.123:0/95423165 >> 172.21.15.123:0/95423165 conn(0x7ffafc0fba10 msgr2=0x7ffafc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.510+0000 7ffafbe00640 1 -- 172.21.15.123:0/95423165 shutdown_connections 2024-03-24T00:39:05.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.511+0000 7ffafbe00640 1 -- 172.21.15.123:0/95423165 wait complete. 2024-03-24T00:39:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 Processor -- start 2024-03-24T00:39:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 -- start start 2024-03-24T00:39:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 0x7ffafc199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 0x7ffafc19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10d260 con 0x7ffafc104930 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10d0e0 con 0x7ffafc107260 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.512+0000 7ffafbe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10d3e0 con 0x7ffafc102000 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 0x7ffafc19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 0x7ffafc19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33910/0 (socket says 172.21.15.123:33910) 2024-03-24T00:39:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45972/0 (socket says 172.21.15.123:45972) 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafa000640 1 -- 172.21.15.123:0/3294770177 learned_addr learned my addr 172.21.15.123:0/3294770177 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafaa00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 0x7ffafc199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 -- 172.21.15.123:0/3294770177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 msgr2=0x7ffafc199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 0x7ffafc199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 -- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 msgr2=0x7ffafc19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 0x7ffafc19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.513+0000 7ffafb400640 1 -- 172.21.15.123:0/3294770177 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc1a1710 con 0x7ffafc107260 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffafb400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009fd0 tx=0x7ffaf400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffaf3400640 1 -- 172.21.15.123:0/3294770177 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400ce20 con 0x7ffafc107260 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffaf3400640 1 -- 172.21.15.123:0/3294770177 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaf4005020 con 0x7ffafc107260 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc1a1a00 con 0x7ffafc107260 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffaf3400640 1 -- 172.21.15.123:0/3294770177 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4010720 con 0x7ffafc107260 2024-03-24T00:39:05.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.514+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc1a1f20 con 0x7ffafc107260 2024-03-24T00:39:05.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.518+0000 7ffaf3400640 1 -- 172.21.15.123:0/3294770177 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffaf4004b20 con 0x7ffafc107260 2024-03-24T00:39:05.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.519+0000 7ffb043ca740 1 Processor -- start 2024-03-24T00:39:05.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.519+0000 7ffb043ca740 1 -- start start 2024-03-24T00:39:05.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44c860 0x557ccc44cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc4523c0 0x557ccc454850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc3a9250 con 0x557ccc4523c0 2024-03-24T00:39:05.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc351a30 con 0x557ccc44c860 2024-03-24T00:39:05.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffb043ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc3a52f0 con 0x557ccc44d250 2024-03-24T00:39:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffafa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44c860 0x557ccc44cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffafaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffafaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49044/0 (socket says 172.21.15.123:49044) 2024-03-24T00:39:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.520+0000 7ffafaa00640 1 -- 172.21.15.123:0/3241390691 learned_addr learned my addr 172.21.15.123:0/3241390691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafa000640 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc4523c0 0x557ccc454850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffaf3400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 0x7ffae00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafaa00640 1 -- 172.21.15.123:0/3241390691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44c860 msgr2=0x557ccc44cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafaa00640 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44c860 0x557ccc44cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafaa00640 1 -- 172.21.15.123:0/3241390691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc4523c0 msgr2=0x557ccc454850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafaa00640 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc4523c0 0x557ccc454850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafaa00640 1 -- 172.21.15.123:0/3241390691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ccc33c690 con 0x557ccc44d250 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.521+0000 7ffafb400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 0x7ffae00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffafaa00640 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00bcb0 tx=0x7ffaec00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5212040e7149f279 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffae7400640 1 -- 172.21.15.123:0/3241390691 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec015070 con 0x557ccc44d250 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffaf3400640 1 -- 172.21.15.123:0/3294770177 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffaf40cf1c0 con 0x7ffafc107260 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffafb400640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 0x7ffae00a4dc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffaf409b810 tx=0x7ffaf4097710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffae7400640 1 -- 172.21.15.123:0/3241390691 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec011920 con 0x557ccc44d250 2024-03-24T00:39:05.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.522+0000 7ffae7400640 1 -- 172.21.15.123:0/3241390691 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0108d0 con 0x557ccc44d250 2024-03-24T00:39:05.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.523+0000 7ffb043ca740 1 -- 172.21.15.123:0/3241390691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 msgr2=0x557ccc451d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.523+0000 7ffb043ca740 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00bcb0 tx=0x7ffaec00f730 comp rx=0 tx=0).stop 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 -- 172.21.15.123:0/3241390691 shutdown_connections 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc4523c0 0x557ccc454850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44d250 0x557ccc451d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 --2- 172.21.15.123:0/3241390691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44c860 0x557ccc44cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 -- 172.21.15.123:0/3241390691 >> 172.21.15.123:0/3241390691 conn(0x557ccc40c200 msgr2=0x557ccc4481d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.524+0000 7ffb043ca740 1 -- 172.21.15.123:0/3241390691 shutdown_connections 2024-03-24T00:39:05.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.525+0000 7ffb043ca740 1 -- 172.21.15.123:0/3241390691 wait complete. 2024-03-24T00:39:05.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.525+0000 7ffb043ca740 1 -- start start 2024-03-24T00:39:05.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.525+0000 7ffb043ca740 1 Processor -- start 2024-03-24T00:39:05.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 0x557ccc456d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc455370 con 0x557ccc44c860 2024-03-24T00:39:05.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc364e80 con 0x557ccc44d250 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ccc4554f0 con 0x557ccc4523c0 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.526+0000 7ffb043ca740 10 client.0 _ll_register_callbacks cb 0x557ccc377130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafa000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33926/0 (socket says 172.21.15.123:33926) 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafa000640 1 -- 172.21.15.123:0/212507595 learned_addr learned my addr 172.21.15.123:0/212507595 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 0x557ccc456d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafa000640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 -- 172.21.15.123:0/212507595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 msgr2=0x557ccc45ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 -- 172.21.15.123:0/212507595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 msgr2=0x557ccc456840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 -- 172.21.15.123:0/212507595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ccc1877d0 con 0x557ccc44d250 2024-03-24T00:39:05.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafa000640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.527+0000 7ffafaa00640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 0x557ccc456d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x557ccc449120 tx=0x7ffaec00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade48fe1b9b4bb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffadbe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:05.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec028060 con 0x557ccc44d250 2024-03-24T00:39:05.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffaec011cd0 con 0x557ccc44d250 2024-03-24T00:39:05.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaec011920 con 0x557ccc44d250 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaec015040 con 0x557ccc44d250 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffb043ca740 10 client.25475 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.528+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557ccc0bd3f0 con 0x557ccc44d250 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec011920 con 0x557ccc44d250 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ffaec022030 con 0x557ccc44d250 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 10 client.25475 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 20 client.25475 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4622b0 0x557ccc4646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 4 client.25475 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.529+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ccc3b6790 con 0x557ccc4622b0 2024-03-24T00:39:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.530+0000 7ffafa000640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4622b0 0x557ccc4646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.530+0000 7ffafa000640 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4622b0 0x557ccc4646a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98be55e30cfa64d6 server_cookie=2254cc9c51379f9b in_seq=0 out_seq=0 2024-03-24T00:39:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.530+0000 7ffadbe00640 10 client.25475 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.531+0000 7ffadbe00640 1 -- 172.21.15.123:0/212507595 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557ccc3b6790 con 0x557ccc4622b0 2024-03-24T00:39:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.531+0000 7ffadbe00640 10 client.25475 handle_command_reply: tid=0 2024-03-24T00:39:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.531+0000 7ffb043ca740 1 client.25475 shutdown 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 msgr2=0x557ccc456d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 0x557ccc456d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x557ccc449120 tx=0x7ffaec00bfd0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 shutdown_connections 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4622b0 0x557ccc4646a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44d250 0x557ccc456d80 secure :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0x557ccc449120 tx=0x7ffaec00bfd0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 --2- 172.21.15.123:0/212507595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc456840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 >> 172.21.15.123:0/212507595 conn(0x557ccc40c200 msgr2=0x557ccc43ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffadbe00640 0 client.25475 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.532+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 shutdown_connections 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.533+0000 7ffb043ca740 1 -- 172.21.15.123:0/212507595 wait complete. 2024-03-24T00:39:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.533+0000 7ffb043ca740 20 client.25475 trim_cache size 0 max 16384 2024-03-24T00:39:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 Processor -- start 2024-03-24T00:39:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 -- start start 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc451870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44b410 0x557ccc44b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a4760 con 0x557ccc44c860 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a6d20 con 0x557ccc4523c0 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.565+0000 7ffb043ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a4d60 con 0x557ccc44b410 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45992/0 (socket says 172.21.15.123:45992) 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 -- 172.21.15.123:0/2122613070 learned_addr learned my addr 172.21.15.123:0/2122613070 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafa000640 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc451870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 -- 172.21.15.123:0/2122613070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44b410 msgr2=0x557ccc44b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44b410 0x557ccc44b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 -- 172.21.15.123:0/2122613070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 msgr2=0x557ccc451870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc451870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 -- 172.21.15.123:0/2122613070 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ccc07d0f0 con 0x557ccc4523c0 2024-03-24T00:39:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.566+0000 7ffafaa00640 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000c00 tx=0x7ffaec005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a00e2e7b05e511ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.567+0000 7ffae6a00640 1 -- 172.21.15.123:0/2122613070 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec028050 con 0x557ccc4523c0 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.567+0000 7ffae6a00640 1 -- 172.21.15.123:0/2122613070 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec004030 con 0x557ccc4523c0 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.567+0000 7ffae6a00640 1 -- 172.21.15.123:0/2122613070 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec032080 con 0x557ccc4523c0 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- 172.21.15.123:0/2122613070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 msgr2=0x557ccc45dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000c00 tx=0x7ffaec005fd0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- 172.21.15.123:0/2122613070 shutdown_connections 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc44b410 0x557ccc44b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc4523c0 0x557ccc45dd80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 --2- 172.21.15.123:0/2122613070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc451870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- 172.21.15.123:0/2122613070 >> 172.21.15.123:0/2122613070 conn(0x557ccc40c200 msgr2=0x557ccc44e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- 172.21.15.123:0/2122613070 shutdown_connections 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- 172.21.15.123:0/2122613070 wait complete. 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.568+0000 7ffb043ca740 1 -- start start 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 Processor -- start 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc45dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a53a0 con 0x557ccc44c860 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a66c0 con 0x557ccc44b410 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1a5a00 con 0x557ccc4523c0 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffb043ca740 10 client.0 _ll_register_callbacks cb 0x7ffae8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.569+0000 7ffafa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46000/0 (socket says 172.21.15.123:46000) 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 -- 172.21.15.123:0/255779666 learned_addr learned my addr 172.21.15.123:0/255779666 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafaa00640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc45dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 -- 172.21.15.123:0/255779666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 msgr2=0x557ccc45bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 -- 172.21.15.123:0/255779666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 msgr2=0x557ccc45dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc45dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 -- 172.21.15.123:0/255779666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ccc45c2b0 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffafa000640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffae802c8f0 tx=0x7ffae802bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26861d30c3b4ec55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffadbe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae804d250 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.570+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffae80046c0 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaec004030 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffae80048c0 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 10 client.25481 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557ccc4be3b0 con 0x557ccc44b410 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaec004030 con 0x557ccc44b410 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ffae804d5b0 con 0x557ccc44b410 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 10 client.25481 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 10 client.25481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 20 client.25481 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4bec70 0x557ccc4c1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 4 client.25481 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:39:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.571+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557ccc3b0ae0 con 0x557ccc4bec70 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.572+0000 7ffafaa00640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4bec70 0x557ccc4c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.573+0000 7ffafaa00640 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4bec70 0x557ccc4c1090 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb2259ab4bffe8d server_cookie=d186ca1d0a66aacb in_seq=0 out_seq=0 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.573+0000 7ffadbe00640 10 client.25481 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffadbe00640 1 -- 172.21.15.123:0/255779666 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557ccc3b0ae0 con 0x557ccc4bec70 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffadbe00640 10 client.25481 handle_command_reply: tid=0 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 client.25481 shutdown 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 msgr2=0x557ccc45d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffae802c8f0 tx=0x7ffae802bc40 comp rx=0 tx=0).stop 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 shutdown_connections 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ccc4bec70 0x557ccc4c1090 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ccc4523c0 0x557ccc45bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ccc44c860 0x557ccc45dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 --2- 172.21.15.123:0/255779666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ccc44b410 0x557ccc45d590 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 >> 172.21.15.123:0/255779666 conn(0x557ccc40c200 msgr2=0x557ccc43e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffadbe00640 0 client.25481 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 shutdown_connections 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 1 -- 172.21.15.123:0/255779666 wait complete. 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.574+0000 7ffb043ca740 20 client.25481 trim_cache size 0 max 16384 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "77394ba9-ece8-41c2-b3c4-da5f5090ad13": { 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "77394ba9-ece8-41c2-b3c4-da5f5090ad13", 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:39:05.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 msgr2=0x7ffae00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 0x7ffae00a4dc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffaf409b810 tx=0x7ffaf4097710 comp rx=0 tx=0).stop 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 msgr2=0x7ffafc1a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009fd0 tx=0x7ffaf400ef10 comp rx=0 tx=0).stop 2024-03-24T00:39:05.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 shutdown_connections 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffafc107260 0x7ffafc1a10a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ffae00a2900 0x7ffae00a4dc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffafc104930 0x7ffafc19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 --2- 172.21.15.123:0/3294770177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffafc102000 0x7ffafc199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.575+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 >> 172.21.15.123:0/3294770177 conn(0x7ffafc0fba10 msgr2=0x7ffafc1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.576+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 shutdown_connections 2024-03-24T00:39:05.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:05.576+0000 7ffafbe00640 1 -- 172.21.15.123:0/3294770177 wait complete. 2024-03-24T00:39:05.596 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:05 smithi123 ceph-mon[31487]: pgmap v421: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:05.602 INFO:tasks.cephfs.filesystem:scrub status for tag:77394ba9-ece8-41c2-b3c4-da5f5090ad13 - {'path': '/', 'tag': '77394ba9-ece8-41c2-b3c4-da5f5090ad13', 'options': 'recursive,force'} 2024-03-24T00:39:05.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:05 smithi177 ceph-mon[36034]: pgmap v421: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 -- 172.21.15.123:0/2502659142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd4075680 msgr2=0x7fafd4073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 --2- 172.21.15.123:0/2502659142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd4075680 0x7fafd4073ad0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009900 tx=0x7fafc402bcc0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 -- 172.21.15.123:0/2502659142 shutdown_connections 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 --2- 172.21.15.123:0/2502659142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd41093b0 0x7fafd410b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 --2- 172.21.15.123:0/2502659142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd40740a0 0x7fafd4074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 --2- 172.21.15.123:0/2502659142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd4075680 0x7fafd4073ad0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 -- 172.21.15.123:0/2502659142 >> 172.21.15.123:0/2502659142 conn(0x7fafd40fcab0 msgr2=0x7fafd40feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 -- 172.21.15.123:0/2502659142 shutdown_connections 2024-03-24T00:39:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.784+0000 7fafd9800640 1 -- 172.21.15.123:0/2502659142 wait complete. 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.785+0000 7fafd9800640 1 Processor -- start 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.785+0000 7fafd9800640 1 -- start start 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4075680 0x7fafd4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd41093b0 0x7fafd41a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd410f3e0 con 0x7fafd4075680 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd410f260 con 0x7fafd40740a0 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd9800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd410f560 con 0x7fafd41093b0 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46012/0 (socket says 172.21.15.123:46012) 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 -- 172.21.15.123:0/211207764 learned_addr learned my addr 172.21.15.123:0/211207764 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd8e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd41093b0 0x7fafd41a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3400640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4075680 0x7fafd4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 -- 172.21.15.123:0/211207764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd41093b0 msgr2=0x7fafd41a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd41093b0 0x7fafd41a0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 -- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4075680 msgr2=0x7fafd4199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4075680 0x7fafd4199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.786+0000 7fafd3e00640 1 -- 172.21.15.123:0/211207764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd41a1630 con 0x7fafd40740a0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd3e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005ab0 tx=0x7fafc402c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd0c00640 1 -- 172.21.15.123:0/211207764 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4043020 con 0x7fafd40740a0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd0c00640 1 -- 172.21.15.123:0/211207764 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafc40054e0 con 0x7fafd40740a0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd0c00640 1 -- 172.21.15.123:0/211207764 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4003cc0 con 0x7fafd40740a0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd41a18c0 con 0x7fafd40740a0 2024-03-24T00:39:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.787+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd41a1dc0 con 0x7fafd40740a0 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.790+0000 7fafdda16740 1 Processor -- start 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.790+0000 7fafdda16740 1 -- start start 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.790+0000 7fafdda16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafdda16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c6880248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafdda16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c688024f50 0x55c6880273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafdda16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c687fe23b0 con 0x55c688024f50 2024-03-24T00:39:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafdda16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c687f84830 con 0x55c68801fde0 2024-03-24T00:39:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafdda16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c687f2d5d0 con 0x55c68801f3f0 2024-03-24T00:39:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49110/0 (socket says 172.21.15.123:49110) 2024-03-24T00:39:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 -- 172.21.15.123:0/1885184033 learned_addr learned my addr 172.21.15.123:0/1885184033 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd3400640 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c688024f50 0x55c6880273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 -- 172.21.15.123:0/1885184033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 msgr2=0x55c6880248e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c6880248e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 -- 172.21.15.123:0/1885184033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c688024f50 msgr2=0x55c6880273e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c688024f50 0x55c6880273e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd8e00640 1 -- 172.21.15.123:0/1885184033 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c687f452c0 con 0x55c68801f3f0 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.791+0000 7fafd0c00640 1 -- 172.21.15.123:0/211207764 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fafc4046020 con 0x7fafd40740a0 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.792+0000 7fafd8e00640 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009df0 tx=0x7fafcc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dab4a8542a5f56c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.792+0000 7fafc9600640 1 -- 172.21.15.123:0/1885184033 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc003970 con 0x55c68801f3f0 2024-03-24T00:39:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.792+0000 7fafc9600640 1 -- 172.21.15.123:0/1885184033 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafcc004030 con 0x55c68801f3f0 2024-03-24T00:39:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.792+0000 7fafc9600640 1 -- 172.21.15.123:0/1885184033 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc011820 con 0x55c68801f3f0 2024-03-24T00:39:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- 172.21.15.123:0/1885184033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 msgr2=0x55c68801f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009df0 tx=0x7fafcc00f730 comp rx=0 tx=0).stop 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- 172.21.15.123:0/1885184033 shutdown_connections 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c688024f50 0x55c6880273e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c6880248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 --2- 172.21.15.123:0/1885184033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c68801f810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- 172.21.15.123:0/1885184033 >> 172.21.15.123:0/1885184033 conn(0x55c68801a960 msgr2=0x55c68801ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- 172.21.15.123:0/1885184033 shutdown_connections 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- 172.21.15.123:0/1885184033 wait complete. 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.793+0000 7fafdda16740 1 -- start start 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafd0c00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafdda16740 1 Processor -- start 2024-03-24T00:39:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafd0c00640 1 -- 172.21.15.123:0/211207764 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fafc40f0c90 con 0x7fafd40740a0 2024-03-24T00:39:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafd8e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafdda16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafdda16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 0x55c688029b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafdda16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c688030b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.794+0000 7fafdda16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c687f40a20 con 0x55c68801fde0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafdda16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c687f808d0 con 0x55c688024f50 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafdda16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c688027f00 con 0x55c68801f3f0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafdda16740 10 client.0 _ll_register_callbacks cb 0x55c687f7d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49116/0 (socket says 172.21.15.123:49116) 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 -- 172.21.15.123:0/778806211 learned_addr learned my addr 172.21.15.123:0/778806211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3400640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 0x55c688029b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd8e00640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c688030b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 -- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 msgr2=0x55c688030b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c688030b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 -- 172.21.15.123:0/778806211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 msgr2=0x55c688029b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 0x55c688029b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3e00640 1 -- 172.21.15.123:0/778806211 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c687c99210 con 0x55c68801f3f0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd8e00640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c688030b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd8e00640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 0x7fafb80a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00fc30 tx=0x7fafcc004740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.795+0000 7fafd3400640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 0x55c688029b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafd3e00640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc40bd3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb9220437ec2ab9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafbe000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc40bcf40 con 0x55c68801f3f0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fafc40bd840 con 0x55c68801f3f0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafcc004030 con 0x55c68801f3f0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafc4003e60 con 0x55c68801f3f0 2024-03-24T00:39:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafdda16740 10 client.25480 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.796+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c687d65620 con 0x55c68801f3f0 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafcc004030 con 0x55c68801f3f0 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fafc40c58f0 con 0x55c68801f3f0 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 10 client.25480 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 20 client.25480 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880353a0 0x55c6880377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 4 client.25480 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c687f91d70 con 0x55c6880353a0 2024-03-24T00:39:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.797+0000 7fafd3400640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880353a0 0x55c6880377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.798+0000 7fafd3400640 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880353a0 0x55c6880377c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a115e7e045ff592c server_cookie=e913f09c93e763f4 in_seq=0 out_seq=0 2024-03-24T00:39:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.798+0000 7fafbe000640 10 client.25480 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafbe000640 1 -- 172.21.15.123:0/778806211 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c687f91d70 con 0x55c6880353a0 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafbe000640 10 client.25480 handle_command_reply: tid=0 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 client.25480 shutdown 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 msgr2=0x55c6880295d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc40bd3b0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 shutdown_connections 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c688030b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801fde0 0x55c688029b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880353a0 0x55c6880377c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafbe000640 0 client.25480 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 --2- 172.21.15.123:0/778806211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801f3f0 0x55c6880295d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 >> 172.21.15.123:0/778806211 conn(0x55c68801a960 msgr2=0x55c688019b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 shutdown_connections 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 1 -- 172.21.15.123:0/778806211 wait complete. 2024-03-24T00:39:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.799+0000 7fafdda16740 20 client.25480 trim_cache size 0 max 16384 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 Processor -- start 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 -- start start 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68803c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 0x55c68803cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41a8b50 con 0x55c68801f3f0 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41ab110 con 0x55c68801fde0 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafdda16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41a9150 con 0x55c688024f50 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafd3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafd3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46028/0 (socket says 172.21.15.123:46028) 2024-03-24T00:39:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.814+0000 7fafd3400640 1 -- 172.21.15.123:0/884062159 learned_addr learned my addr 172.21.15.123:0/884062159 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3e00640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68803c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd8e00640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 0x55c68803cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 -- 172.21.15.123:0/884062159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 msgr2=0x55c68803cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 0x55c68803cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 -- 172.21.15.123:0/884062159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 msgr2=0x55c68803c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68803c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 -- 172.21.15.123:0/884062159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c688037b30 con 0x55c68801fde0 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd8e00640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 0x55c68803cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3e00640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68803c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafd3400640 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fafc0000c00 tx=0x7fafc001eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bd716a2bcc36955 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafc8c00640 1 -- 172.21.15.123:0/884062159 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc001e050 con 0x55c68801fde0 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafc8c00640 1 -- 172.21.15.123:0/884062159 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafc001e1f0 con 0x55c68801fde0 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.815+0000 7fafc8c00640 1 -- 172.21.15.123:0/884062159 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0012740 con 0x55c68801fde0 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.816+0000 7fafdda16740 1 -- 172.21.15.123:0/884062159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 msgr2=0x55c68803c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.816+0000 7fafdda16740 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fafc0000c00 tx=0x7fafc001eac0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 -- 172.21.15.123:0/884062159 shutdown_connections 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c688024f50 0x55c68803cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c68801fde0 0x55c68803c5c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 --2- 172.21.15.123:0/884062159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68803c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 -- 172.21.15.123:0/884062159 >> 172.21.15.123:0/884062159 conn(0x55c68801a960 msgr2=0x55c68801dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 -- 172.21.15.123:0/884062159 shutdown_connections 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 -- 172.21.15.123:0/884062159 wait complete. 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 -- start start 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.817+0000 7fafdda16740 1 Processor -- start 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68802dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801fde0 0x55c68802e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41a9790 con 0x55c68801f3f0 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41aaab0 con 0x55c688024f50 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41a9df0 con 0x55c68801fde0 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafdda16740 10 client.0 _ll_register_callbacks cb 0x7fafc4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafd8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafd8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46044/0 (socket says 172.21.15.123:46044) 2024-03-24T00:39:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.818+0000 7fafd8e00640 1 -- 172.21.15.123:0/2582122409 learned_addr learned my addr 172.21.15.123:0/2582122409 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd3e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68802dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd3400640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801fde0 0x55c68802e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 -- 172.21.15.123:0/2582122409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801fde0 msgr2=0x55c68802e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801fde0 0x55c68802e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 -- 172.21.15.123:0/2582122409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 msgr2=0x55c68802dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68802dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 -- 172.21.15.123:0/2582122409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c68802f850 con 0x55c688024f50 2024-03-24T00:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd3e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68802dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:05 smithi204 ceph-mon[36345]: pgmap v421: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafd8e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fafcc018000 tx=0x7fafcc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07bf80253c26efa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.819+0000 7fafbe000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc036060 con 0x55c688024f50 2024-03-24T00:39:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafc001e1f0 con 0x55c688024f50 2024-03-24T00:39:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fafcc003b80 con 0x55c688024f50 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafcc003d80 con 0x55c688024f50 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafdda16740 10 client.25493 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.820+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c688090fc0 con 0x55c688024f50 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafc001e1f0 con 0x55c688024f50 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fafcc005040 con 0x55c688024f50 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 10 client.25493 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 10 client.25493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 20 client.25493 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880916c0 0x55c688093b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 4 client.25493 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c687f8c0c0 con 0x55c6880916c0 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.821+0000 7fafd3e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880916c0 0x55c688093b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.822+0000 7fafd3e00640 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880916c0 0x55c688093b20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba24e055ad77a6fd server_cookie=6b7f3b09adfea3c1 in_seq=0 out_seq=0 2024-03-24T00:39:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.822+0000 7fafbe000640 10 client.25493 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.826+0000 7fafbe000640 1 -- 172.21.15.123:0/2582122409 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c687f8c0c0 con 0x55c6880916c0 2024-03-24T00:39:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.826+0000 7fafbe000640 10 client.25493 handle_command_reply: tid=0 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 client.25493 shutdown 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 msgr2=0x55c68802f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fafcc018000 tx=0x7fafcc00f730 comp rx=0 tx=0).stop 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 shutdown_connections 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c688024f50 0x55c68802f260 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c6880916c0 0x55c688093b20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c68801fde0 0x55c68802e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 --2- 172.21.15.123:0/2582122409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c68801f3f0 0x55c68802dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 >> 172.21.15.123:0/2582122409 conn(0x55c68801a960 msgr2=0x55c68801dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafbe000640 0 client.25493 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 shutdown_connections 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 1 -- 172.21.15.123:0/2582122409 wait complete. 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.827+0000 7fafdda16740 20 client.25493 trim_cache size 0 max 16384 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:39:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 msgr2=0x7fafb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 0x7fafb80a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00fc30 tx=0x7fafcc004740 comp rx=0 tx=0).stop 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 msgr2=0x7fafd4199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005ab0 tx=0x7fafc402c750 comp rx=0 tx=0).stop 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 shutdown_connections 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafd41093b0 0x7fafd41a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4075680 0x7fafd4199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 --2- 172.21.15.123:0/211207764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd40740a0 0x7fafd4199a10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.828+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 >> 172.21.15.123:0/211207764 conn(0x7fafd40fcab0 msgr2=0x7fafd4105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.829+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 shutdown_connections 2024-03-24T00:39:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:05.829+0000 7fafd9800640 1 -- 172.21.15.123:0/211207764 wait complete. 2024-03-24T00:39:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:06 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:39:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:06 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:06 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:39:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:06 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:06 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:39:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:06 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:06 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:39:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:06 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:06 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:39:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:06 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:06 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:39:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:06 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:07 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:39:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:07 smithi123 ceph-mon[31487]: pgmap v422: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T00:39:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:07 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:39:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:07 smithi177 ceph-mon[36034]: pgmap v422: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T00:39:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:07 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:39:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:07 smithi204 ceph-mon[36345]: pgmap v422: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T00:39:10.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:10 smithi177 ceph-mon[36034]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T00:39:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:10 smithi204 ceph-mon[36345]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T00:39:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:10 smithi123 ceph-mon[31487]: pgmap v423: 97 pgs: 97 active+clean; 664 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T00:39:12.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:12 smithi177 ceph-mon[36034]: pgmap v424: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:12 smithi204 ceph-mon[36345]: pgmap v424: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:12 smithi123 ceph-mon[31487]: pgmap v424: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:14 smithi204 ceph-mon[36345]: pgmap v425: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:14 smithi123 ceph-mon[31487]: pgmap v425: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:14 smithi177 ceph-mon[36034]: pgmap v425: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:16 smithi204 ceph-mon[36345]: pgmap v426: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:16 smithi123 ceph-mon[31487]: pgmap v426: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:16 smithi177 ceph-mon[36034]: pgmap v426: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:18.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:18 smithi204 ceph-mon[36345]: pgmap v427: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:18 smithi123 ceph-mon[31487]: pgmap v427: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:18 smithi177 ceph-mon[36034]: pgmap v427: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 -- 172.21.15.123:0/147357335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac104a90 msgr2=0x7f3dac106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 --2- 172.21.15.123:0/147357335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac104a90 0x7f3dac106e80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009900 tx=0x7f3d9402bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 -- 172.21.15.123:0/147357335 shutdown_connections 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 --2- 172.21.15.123:0/147357335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac1073c0 0x7f3dac109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 --2- 172.21.15.123:0/147357335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac104a90 0x7f3dac106e80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 --2- 172.21.15.123:0/147357335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac102160 0x7f3dac104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 -- 172.21.15.123:0/147357335 >> 172.21.15.123:0/147357335 conn(0x7f3dac0fbb50 msgr2=0x7f3dac0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 -- 172.21.15.123:0/147357335 shutdown_connections 2024-03-24T00:39:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.963+0000 7f3db2800640 1 -- 172.21.15.123:0/147357335 wait complete. 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.964+0000 7f3db2800640 1 Processor -- start 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.964+0000 7f3db2800640 1 -- start start 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac104a90 0x7f3dac197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 0x7f3dac19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10d3b0 con 0x7f3dac104a90 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10d230 con 0x7f3dac1073c0 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3db2800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10d530 con 0x7f3dac102160 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3da3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac104a90 0x7f3dac197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:55188/0 (socket says 172.21.15.123:55188) 2024-03-24T00:39:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 0x7f3dac19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 -- 172.21.15.123:0/3940355749 learned_addr learned my addr 172.21.15.123:0/3940355749 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 -- 172.21.15.123:0/3940355749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 msgr2=0x7f3dac19eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 0x7f3dac19eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 -- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac104a90 msgr2=0x7f3dac197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac104a90 0x7f3dac197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dab400640 1 -- 172.21.15.123:0/3940355749 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac19f540 con 0x7f3dac102160 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.965+0000 7f3dabe00640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 0x7f3dac19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3dab400640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00ea60 tx=0x7f3d9c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3da8c00640 1 -- 172.21.15.123:0/3940355749 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00cde0 con 0x7f3dac102160 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dac19f830 con 0x7f3dac102160 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3da8c00640 1 -- 172.21.15.123:0/3940355749 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9c0035f0 con 0x7f3dac102160 2024-03-24T00:39:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3da8c00640 1 -- 172.21.15.123:0/3940355749 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c010670 con 0x7f3dac102160 2024-03-24T00:39:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.966+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac19fda0 con 0x7f3dac102160 2024-03-24T00:39:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.970+0000 7f3db55a2740 1 Processor -- start 2024-03-24T00:39:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.970+0000 7f3db55a2740 1 -- start start 2024-03-24T00:39:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.970+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.970+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d396d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d3c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3db55a2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5cf73b0 con 0x5573b5d39d40 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3db55a2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5c99830 con 0x5573b5d341e0 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3db55a2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5c425d0 con 0x5573b5d34bd0 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3da8c00640 1 -- 172.21.15.123:0/3940355749 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3d9c0037a0 con 0x7f3dac102160 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52964/0 (socket says 172.21.15.123:52964) 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 -- 172.21.15.123:0/1120461662 learned_addr learned my addr 172.21.15.123:0/1120461662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3da3e00640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d3c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dab400640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 -- 172.21.15.123:0/1120461662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 msgr2=0x5573b5d396d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d396d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 -- 172.21.15.123:0/1120461662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 msgr2=0x5573b5d3c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d3c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 -- 172.21.15.123:0/1120461662 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573b5c5a2c0 con 0x5573b5d341e0 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3da3e00640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d3c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dab400640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d396d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.971+0000 7f3dabe00640 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3da40117e0 tx=0x7f3da4011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b807366a62f1043e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3da0c00640 1 -- 172.21.15.123:0/1120461662 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4011310 con 0x5573b5d341e0 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3da0c00640 1 -- 172.21.15.123:0/1120461662 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3da4003e50 con 0x5573b5d341e0 2024-03-24T00:39:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3da0c00640 1 -- 172.21.15.123:0/1120461662 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4004620 con 0x5573b5d341e0 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3da8c00640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 0x7f3d900a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3da8c00640 1 -- 172.21.15.123:0/3940355749 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d9c0ced40 con 0x7f3dac102160 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3db55a2740 1 -- 172.21.15.123:0/1120461662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 msgr2=0x5573b5d34600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3db55a2740 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3da40117e0 tx=0x7f3da4011ca0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.972+0000 7f3dab400640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 0x7f3d900a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 -- 172.21.15.123:0/1120461662 shutdown_connections 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d3c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d396d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 --2- 172.21.15.123:0/1120461662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d34600 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 -- 172.21.15.123:0/1120461662 >> 172.21.15.123:0/1120461662 conn(0x5573b5d2f750 msgr2=0x5573b5d2fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 -- 172.21.15.123:0/1120461662 shutdown_connections 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 -- 172.21.15.123:0/1120461662 wait complete. 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3dab400640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 0x7f3d900a4d60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0ccbc0 tx=0x7f3d9c00ef90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 -- start start 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.973+0000 7f3db55a2740 1 Processor -- start 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 0x5573b5d3e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d3e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 0x5573b5d45930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5c55a20 con 0x5573b5d341e0 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5c958d0 con 0x5573b5d39d40 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b5d3ccf0 con 0x5573b5d34bd0 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3db55a2740 10 client.0 _ll_register_callbacks cb 0x5573b5c921c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 0x5573b5d3e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d3e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 0x5573b5d3e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55438/0 (socket says 172.21.15.123:55438) 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3dabe00640 1 -- 172.21.15.123:0/2791349144 learned_addr learned my addr 172.21.15.123:0/2791349144 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 0x5573b5d45930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 -- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 msgr2=0x5573b5d45930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 0x5573b5d45930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 -- 172.21.15.123:0/2791349144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 msgr2=0x5573b5d3e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 0x5573b5d3e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3da3e00640 1 -- 172.21.15.123:0/2791349144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573b59ae210 con 0x5573b5d34bd0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.974+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 0x5573b5d45930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.975+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d3e960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009230 tx=0x7f3d940042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57538396fa7baed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.975+0000 7f3d99600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.975+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005d50 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.975+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d940035f0 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.975+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3da4003e50 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d940044e0 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 10 client.25492 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5573b5a7a620 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3da4003e50 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3d9404a890 con 0x5573b5d34bd0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 10 client.25492 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 10 client.25492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 20 client.25492 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5d4a110 0x5573b5d4c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 4 client.25492 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.976+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573b5ca6d70 con 0x5573b5d4a110 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.977+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5d4a110 0x5573b5d4c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.977+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5d4a110 0x5573b5d4c500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2edf0e1db0b3e067 server_cookie=e152ca7dff90bbaf in_seq=0 out_seq=0 2024-03-24T00:39:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.977+0000 7f3d99600640 10 client.25492 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3d99600640 1 -- 172.21.15.123:0/2791349144 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5573b5ca6d70 con 0x5573b5d4a110 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3d99600640 10 client.25492 handle_command_reply: tid=0 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 client.25492 shutdown 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 msgr2=0x5573b5d3e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d3e960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009230 tx=0x7f3d940042f0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 shutdown_connections 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d39d40 0x5573b5d45930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5d4a110 0x5573b5d4c500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d3e960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2791349144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d341e0 0x5573b5d3e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3d99600640 0 client.25492 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 >> 172.21.15.123:0/2791349144 conn(0x5573b5d2f750 msgr2=0x5573b5d2e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 shutdown_connections 2024-03-24T00:39:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 1 -- 172.21.15.123:0/2791349144 wait complete. 2024-03-24T00:39:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:18.978+0000 7f3db55a2740 20 client.25492 trim_cache size 0 max 16384 2024-03-24T00:39:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 Processor -- start 2024-03-24T00:39:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 -- start start 2024-03-24T00:39:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d2e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d2e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a6a40 con 0x5573b5d39d40 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a9000 con 0x5573b5d341e0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.004+0000 7f3db55a2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a7040 con 0x5573b5d34bd0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d2e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d395c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52992/0 (socket says 172.21.15.123:52992) 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3dabe00640 1 -- 172.21.15.123:0/4245296801 learned_addr learned my addr 172.21.15.123:0/4245296801 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3dabe00640 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d2e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 -- 172.21.15.123:0/4245296801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 msgr2=0x5573b5d395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d395c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 -- 172.21.15.123:0/4245296801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 msgr2=0x5573b5d2e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d2e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 -- 172.21.15.123:0/4245296801 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573b5d4cb60 con 0x5573b5d34bd0 2024-03-24T00:39:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.005+0000 7f3da3e00640 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d2e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9404ace0 tx=0x7f3d94000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda2516cc099d4c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.006+0000 7f3d9be00640 1 -- 172.21.15.123:0/4245296801 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005d50 con 0x5573b5d34bd0 2024-03-24T00:39:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.006+0000 7f3d9be00640 1 -- 172.21.15.123:0/4245296801 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9404ae50 con 0x5573b5d34bd0 2024-03-24T00:39:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.006+0000 7f3d9be00640 1 -- 172.21.15.123:0/4245296801 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003c40 con 0x5573b5d34bd0 2024-03-24T00:39:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- 172.21.15.123:0/4245296801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 msgr2=0x5573b5d2e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d2e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9404ace0 tx=0x7f3d94000bc0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- 172.21.15.123:0/4245296801 shutdown_connections 2024-03-24T00:39:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5d2e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d34bd0 0x5573b5d2e0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 --2- 172.21.15.123:0/4245296801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d341e0 0x5573b5d395c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- 172.21.15.123:0/4245296801 >> 172.21.15.123:0/4245296801 conn(0x5573b5d2f750 msgr2=0x5573b5d35f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- 172.21.15.123:0/4245296801 shutdown_connections 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- 172.21.15.123:0/4245296801 wait complete. 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 -- start start 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.007+0000 7f3db55a2740 1 Processor -- start 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 0x5573b5da1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5da9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a7680 con 0x5573b5d39d40 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a89a0 con 0x5573b5d34bd0 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a7ce0 con 0x5573b5d341e0 2024-03-24T00:39:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3db55a2740 10 client.0 _ll_register_callbacks cb 0x7f3da40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52996/0 (socket says 172.21.15.123:52996) 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3dabe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5da9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 -- 172.21.15.123:0/2446442828 learned_addr learned my addr 172.21.15.123:0/2446442828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 0x5573b5da1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 -- 172.21.15.123:0/2446442828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 msgr2=0x5573b5da1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 0x5573b5da1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 -- 172.21.15.123:0/2446442828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 msgr2=0x5573b5da9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5da9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3da3e00640 1 -- 172.21.15.123:0/2446442828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573b5da9650 con 0x5573b5d34bd0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.008+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 0x5573b5da1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3da3e00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d9404c010 tx=0x7f3d9402c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74def0a55d980077 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005d50 con 0x5573b5d34bd0 2024-03-24T00:39:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d9404a890 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d9404ae50 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d940042f0 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3db55a2740 10 client.25511 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5573b5daa960 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9404ae50 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3d9405d420 con 0x5573b5d34bd0 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3db55a2740 10 client.25511 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.009+0000 7f3db55a2740 10 client.25511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3db55a2740 20 client.25511 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5dab220 0x5573b5dad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3db55a2740 4 client.25511 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5573b5ca10c0 con 0x5573b5dab220 2024-03-24T00:39:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5dab220 0x5573b5dad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3dabe00640 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5dab220 0x5573b5dad640 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc2b5b7fd47786e server_cookie=1d7feffd7eca77b0 in_seq=0 out_seq=0 2024-03-24T00:39:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.010+0000 7f3d99600640 10 client.25511 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3d99600640 1 -- 172.21.15.123:0/2446442828 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5573b5ca10c0 con 0x5573b5dab220 2024-03-24T00:39:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3d99600640 10 client.25511 handle_command_reply: tid=0 2024-03-24T00:39:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 client.25511 shutdown 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 msgr2=0x5573b5da2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d9404c010 tx=0x7f3d9402c880 comp rx=0 tx=0).stop 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 shutdown_connections 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573b5d39d40 0x5573b5da9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5573b5d34bd0 0x5573b5da2020 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5573b5dab220 0x5573b5dad640 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 --2- 172.21.15.123:0/2446442828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5573b5d341e0 0x5573b5da1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3d99600640 0 client.25511 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 >> 172.21.15.123:0/2446442828 conn(0x5573b5d2f750 msgr2=0x5573b5d329b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.015+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 shutdown_connections 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db55a2740 1 -- 172.21.15.123:0/2446442828 wait complete. 2024-03-24T00:39:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db55a2740 20 client.25511 trim_cache size 0 max 16384 2024-03-24T00:39:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:39:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:39:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:39:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 msgr2=0x7f3d900a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 0x7f3d900a4d60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0ccbc0 tx=0x7f3d9c00ef90 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 msgr2=0x7f3dac197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00ea60 tx=0x7f3d9c00ef20 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 shutdown_connections 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac1073c0 0x7f3dac19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d900a28a0 0x7f3d900a4d60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dac104a90 0x7f3dac197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 --2- 172.21.15.123:0/3940355749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dac102160 0x7f3dac197920 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.016+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 >> 172.21.15.123:0/3940355749 conn(0x7f3dac0fbb50 msgr2=0x7f3dac102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.017+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 shutdown_connections 2024-03-24T00:39:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:19.017+0000 7f3db2800640 1 -- 172.21.15.123:0/3940355749 wait complete. 2024-03-24T00:39:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:20 smithi204 ceph-mon[36345]: pgmap v428: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:20 smithi123 ceph-mon[31487]: pgmap v428: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:20 smithi177 ceph-mon[36034]: pgmap v428: 97 pgs: 97 active+clean; 665 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T00:39:21.808 INFO:tasks.workunit.client.0.smithi123.stderr:Updating files: 63% (7790/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-24T00:39:22.553 INFO:tasks.workunit.client.0.smithi123.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T00:39:22.553 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: git switch -c 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:Or undo this operation with: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: git switch - 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.554 INFO:tasks.workunit.client.0.smithi123.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T00:39:22.555 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-24T00:39:22.555 INFO:tasks.workunit.client.0.smithi123.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T00:39:22.563 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T00:39:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:22 smithi204 ceph-mon[36345]: pgmap v429: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T00:39:22.611 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:22 smithi123 ceph-mon[31487]: pgmap v429: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T00:39:22.667 INFO:tasks.workunit.client.0.smithi123.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T00:39:22.675 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:39:22.675 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T00:39:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:22 smithi177 ceph-mon[36034]: pgmap v429: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T00:39:23.648 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T00:39:23.756 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T00:39:23.836 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:39:23.843 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:39:23.843 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T00:39:23.926 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:39:23.935 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:39:23.935 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T00:39:23.983 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-24T00:39:23.984 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-24T00:39:23.985 DEBUG:teuthology.orchestra.run.smithi123:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-24T00:39:24.103 INFO:tasks.workunit.client.0.smithi123.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-24T00:39:24.106 INFO:tasks.workunit.client.0.smithi123.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-24T00:39:24.106 INFO:tasks.workunit.client.0.smithi123.stderr:+ trap -- retry EXIT 2024-03-24T00:39:24.106 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -rf ffsb 2024-03-24T00:39:24.110 INFO:tasks.workunit.client.0.smithi123.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-24T00:39:24.118 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into 'ffsb'... 2024-03-24T00:39:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:24 smithi204 ceph-mon[36345]: pgmap v430: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:39:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:24 smithi123 ceph-mon[31487]: pgmap v430: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:39:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:24 smithi177 ceph-mon[36034]: pgmap v430: 97 pgs: 97 active+clean; 666 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T00:39:26.146 INFO:tasks.workunit.client.0.smithi123.stderr:+ trap - EXIT 2024-03-24T00:39:26.146 INFO:tasks.workunit.client.0.smithi123.stderr:+ cd ffsb 2024-03-24T00:39:26.147 INFO:tasks.workunit.client.0.smithi123.stderr:+ ./configure 2024-03-24T00:39:26.466 INFO:tasks.workunit.client.0.smithi123.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T00:39:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:26 smithi204 ceph-mon[36345]: pgmap v431: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 17 op/s 2024-03-24T00:39:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:26 smithi123 ceph-mon[31487]: pgmap v431: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 17 op/s 2024-03-24T00:39:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:26 smithi177 ceph-mon[36034]: pgmap v431: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 17 op/s 2024-03-24T00:39:27.490 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether build environment is sane... yes 2024-03-24T00:39:27.524 INFO:tasks.workunit.client.0.smithi123.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-24T00:39:27.524 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gawk... gawk 2024-03-24T00:39:27.557 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether make sets $(MAKE)... yes 2024-03-24T00:39:27.662 INFO:tasks.workunit.client.0.smithi123.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-24T00:39:27.663 INFO:tasks.workunit.client.0.smithi123.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-24T00:39:27.683 INFO:tasks.workunit.client.0.smithi123.stdout:checking for style of include used by make... GNU 2024-03-24T00:39:27.697 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc... gcc 2024-03-24T00:39:27.838 INFO:tasks.workunit.client.0.smithi123.stdout:checking for C compiler default output file name... a.out 2024-03-24T00:39:27.846 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether the C compiler works... yes 2024-03-24T00:39:27.856 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are cross compiling... no 2024-03-24T00:39:27.942 INFO:tasks.workunit.client.0.smithi123.stdout:checking for suffix of executables... 2024-03-24T00:39:28.029 INFO:tasks.workunit.client.0.smithi123.stdout:checking for suffix of object files... o 2024-03-24T00:39:28.136 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T00:39:28.247 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether gcc accepts -g... yes 2024-03-24T00:39:28.427 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-24T00:39:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:28 smithi204 ceph-mon[36345]: pgmap v432: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-03-24T00:39:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:28 smithi123 ceph-mon[31487]: pgmap v432: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-03-24T00:39:28.689 INFO:tasks.workunit.client.0.smithi123.stdout:checking dependency style of gcc... gcc3 2024-03-24T00:39:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:28 smithi177 ceph-mon[36034]: pgmap v432: 97 pgs: 97 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-03-24T00:39:28.870 INFO:tasks.workunit.client.0.smithi123.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T00:39:29.024 INFO:tasks.workunit.client.0.smithi123.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-24T00:39:29.030 INFO:tasks.workunit.client.0.smithi123.stdout:checking for egrep... /usr/bin/grep -E 2024-03-24T00:39:29.065 INFO:tasks.workunit.client.0.smithi123.stdout:checking for AIX... no 2024-03-24T00:39:29.148 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc... (cached) gcc 2024-03-24T00:39:29.148 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-24T00:39:29.148 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-24T00:39:29.148 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-24T00:39:29.148 INFO:tasks.workunit.client.0.smithi123.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-24T00:39:29.301 INFO:tasks.workunit.client.0.smithi123.stdout:checking for main in -lm... yes 2024-03-24T00:39:29.430 INFO:tasks.workunit.client.0.smithi123.stdout:checking for main in -lpthread... yes 2024-03-24T00:39:29.811 INFO:tasks.workunit.client.0.smithi123.stdout:checking for ANSI C header files... yes 2024-03-24T00:39:29.975 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-24T00:39:30.103 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/types.h... yes 2024-03-24T00:39:30.228 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/stat.h... yes 2024-03-24T00:39:30.365 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdlib.h... yes 2024-03-24T00:39:30.508 INFO:tasks.workunit.client.0.smithi123.stdout:checking for string.h... yes 2024-03-24T00:39:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:30 smithi204 ceph-mon[36345]: pgmap v433: 97 pgs: 97 active+clean; 1.3 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 63 KiB/s wr, 17 op/s 2024-03-24T00:39:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:30 smithi123 ceph-mon[31487]: pgmap v433: 97 pgs: 97 active+clean; 1.3 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 63 KiB/s wr, 17 op/s 2024-03-24T00:39:30.661 INFO:tasks.workunit.client.0.smithi123.stdout:checking for memory.h... yes 2024-03-24T00:39:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:30 smithi177 ceph-mon[36034]: pgmap v433: 97 pgs: 97 active+clean; 1.3 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 63 KiB/s wr, 17 op/s 2024-03-24T00:39:30.795 INFO:tasks.workunit.client.0.smithi123.stdout:checking for strings.h... yes 2024-03-24T00:39:30.933 INFO:tasks.workunit.client.0.smithi123.stdout:checking for inttypes.h... yes 2024-03-24T00:39:31.070 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdint.h... yes 2024-03-24T00:39:31.205 INFO:tasks.workunit.client.0.smithi123.stdout:checking for unistd.h... yes 2024-03-24T00:39:31.349 INFO:tasks.workunit.client.0.smithi123.stdout:checking pthread.h usability... yes 2024-03-24T00:39:31.424 INFO:tasks.workunit.client.0.smithi123.stdout:checking pthread.h presence... yes 2024-03-24T00:39:31.426 INFO:tasks.workunit.client.0.smithi123.stdout:checking for pthread.h... yes 2024-03-24T00:39:31.564 INFO:tasks.workunit.client.0.smithi123.stdout:checking fcntl.h usability... yes 2024-03-24T00:39:31.635 INFO:tasks.workunit.client.0.smithi123.stdout:checking fcntl.h presence... yes 2024-03-24T00:39:31.637 INFO:tasks.workunit.client.0.smithi123.stdout:checking for fcntl.h... yes 2024-03-24T00:39:31.775 INFO:tasks.workunit.client.0.smithi123.stdout:checking limits.h usability... yes 2024-03-24T00:39:31.852 INFO:tasks.workunit.client.0.smithi123.stdout:checking limits.h presence... yes 2024-03-24T00:39:31.854 INFO:tasks.workunit.client.0.smithi123.stdout:checking for limits.h... yes 2024-03-24T00:39:31.879 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdint.h... (cached) yes 2024-03-24T00:39:32.017 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/time.h usability... yes 2024-03-24T00:39:32.103 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/time.h presence... yes 2024-03-24T00:39:32.105 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/time.h... yes 2024-03-24T00:39:32.127 INFO:tasks.workunit.client.0.smithi123.stdout:checking for unistd.h... (cached) yes 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 -- 172.21.15.123:0/3688534925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c841016d0 msgr2=0x7f9c84101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/3688534925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c841016d0 0x7f9c84101af0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009900 tx=0x7f9c7402bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 -- 172.21.15.123:0/3688534925 shutdown_connections 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/3688534925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c84103280 0x7f9c84109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/3688534925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841028c0 0x7f9c84102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/3688534925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c841016d0 0x7f9c84101af0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 -- 172.21.15.123:0/3688534925 >> 172.21.15.123:0/3688534925 conn(0x7f9c840fcc60 msgr2=0x7f9c840ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 -- 172.21.15.123:0/3688534925 shutdown_connections 2024-03-24T00:39:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.150+0000 7f9c8ae00640 1 -- 172.21.15.123:0/3688534925 wait complete. 2024-03-24T00:39:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 Processor -- start 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 -- start start 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 0x7f9c84199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 0x7f9c8419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8410d370 con 0x7f9c841028c0 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8410d1f0 con 0x7f9c84103280 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.151+0000 7f9c8ae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8410d4f0 con 0x7f9c841016d0 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 0x7f9c8419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52922/0 (socket says 172.21.15.123:52922) 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 0x7f9c8419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43874/0 (socket says 172.21.15.123:43874) 2024-03-24T00:39:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 -- 172.21.15.123:0/307134124 learned_addr learned my addr 172.21.15.123:0/307134124 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c83e00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 0x7f9c84199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 -- 172.21.15.123:0/307134124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 msgr2=0x7f9c84199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 0x7f9c84199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 -- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 msgr2=0x7f9c8419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 0x7f9c8419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 -- 172.21.15.123:0/307134124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c841a1770 con 0x7f9c84103280 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c83e00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 0x7f9c84199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c88e00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00eb10 tx=0x7f9c7c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c81600640 1 -- 172.21.15.123:0/307134124 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c004b20 con 0x7f9c84103280 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c81600640 1 -- 172.21.15.123:0/307134124 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c004cc0 con 0x7f9c84103280 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.152+0000 7f9c81600640 1 -- 172.21.15.123:0/307134124 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c010760 con 0x7f9c84103280 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.153+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c841a1a60 con 0x7f9c84103280 2024-03-24T00:39:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.153+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c841a1f60 con 0x7f9c84103280 2024-03-24T00:39:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.157+0000 7f9c8dba7740 1 Processor -- start 2024-03-24T00:39:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.157+0000 7f9c81600640 1 -- 172.21.15.123:0/307134124 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9c7c005020 con 0x7f9c84103280 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.157+0000 7f9c8dba7740 1 -- start start 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574052753b0 con 0x5574052b21e0 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557405217830 con 0x5574052b2bd0 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574051c05d0 con 0x5574052b7d40 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52936/0 (socket says 172.21.15.123:52936) 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 -- 172.21.15.123:0/286197254 learned_addr learned my addr 172.21.15.123:0/286197254 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c7b400640 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c7b400640 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 -- 172.21.15.123:0/286197254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 msgr2=0x5574052ba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052ba1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 -- 172.21.15.123:0/286197254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 msgr2=0x5574052b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.158+0000 7f9c83e00640 1 -- 172.21.15.123:0/286197254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574051d82c0 con 0x5574052b2bd0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c81600640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 0x7f9c680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c83e00640 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000c00 tx=0x7f9c740095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf57d89aa37bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c81600640 1 -- 172.21.15.123:0/307134124 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c7c0cefe0 con 0x7f9c84103280 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c7b400640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 0x7f9c680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c79600640 1 -- 172.21.15.123:0/286197254 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74044020 con 0x5574052b2bd0 2024-03-24T00:39:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c79600640 1 -- 172.21.15.123:0/286197254 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c7402cb80 con 0x5574052b2bd0 2024-03-24T00:39:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.159+0000 7f9c79600640 1 -- 172.21.15.123:0/286197254 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c740046c0 con 0x5574052b2bd0 2024-03-24T00:39:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- 172.21.15.123:0/286197254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 msgr2=0x5574052b76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000c00 tx=0x7f9c740095e0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c7b400640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 0x7f9c680a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x5574052b3430 tx=0x7f9c6c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- 172.21.15.123:0/286197254 shutdown_connections 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052ba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052b76d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/286197254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- 172.21.15.123:0/286197254 >> 172.21.15.123:0/286197254 conn(0x5574052ad750 msgr2=0x5574052adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- 172.21.15.123:0/286197254 shutdown_connections 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- 172.21.15.123:0/286197254 wait complete. 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.160+0000 7f9c8dba7740 1 -- start start 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 Processor -- start 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 0x5574052bc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 0x5574052bc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574051d3a20 con 0x5574052b2bd0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574052138d0 con 0x5574052b7d40 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574052bac90 con 0x5574052b21e0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c8dba7740 10 client.0 _ll_register_callbacks cb 0x5574052101c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 0x5574052bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c88e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52948/0 (socket says 172.21.15.123:52948) 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 0x5574052bc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50450/0 (socket says 172.21.15.123:50450) 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.161+0000 7f9c88e00640 1 -- 172.21.15.123:0/2235222368 learned_addr learned my addr 172.21.15.123:0/2235222368 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c7b400640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 0x5574052bc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 -- 172.21.15.123:0/2235222368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 msgr2=0x5574052bc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 0x5574052bc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 -- 172.21.15.123:0/2235222368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 msgr2=0x5574052bc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 0x5574052bc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 -- 172.21.15.123:0/2235222368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557404f2c210 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c7b400640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 0x5574052bc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c88e00640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0a3c00 tx=0x7f9c7c09b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb378983e9700f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c72000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0b4060 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c7c0a3ea0 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c7402cb80 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.162+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7c09fae0 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 10 client.25523 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557404ff8620 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c7402cb80 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9c7c0860c0 con 0x5574052b7d40 2024-03-24T00:39:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 10 client.25523 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 10 client.25523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 20 client.25523 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5574052c81f0 0x5574052ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 4 client.25523 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557405224d70 con 0x5574052c81f0 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.163+0000 7f9c83e00640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5574052c81f0 0x5574052ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.164+0000 7f9c83e00640 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5574052c81f0 0x5574052ca620 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c51b81ea0d71a650 server_cookie=1b8d5f0ace807818 in_seq=0 out_seq=0 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.164+0000 7f9c72000640 10 client.25523 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c72000640 1 -- 172.21.15.123:0/2235222368 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557405224d70 con 0x5574052c81f0 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c72000640 10 client.25523 handle_command_reply: tid=0 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 client.25523 shutdown 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 msgr2=0x5574052c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0a3c00 tx=0x7f9c7c09b3b0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 shutdown_connections 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b7d40 0x5574052c38e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b2bd0 0x5574052bc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5574052c81f0 0x5574052ca620 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2235222368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b21e0 0x5574052bc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c72000640 0 client.25523 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 >> 172.21.15.123:0/2235222368 conn(0x5574052ad750 msgr2=0x5574052ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 shutdown_connections 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2235222368 wait complete. 2024-03-24T00:39:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.165+0000 7f9c8dba7740 20 client.25523 trim_cache size 0 max 16384 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 Processor -- start 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 -- start start 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b2bd0 0x5574052c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 0x5574052b4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841a8c30 con 0x5574052b7d40 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841ab1f0 con 0x5574052b21e0 2024-03-24T00:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841a9230 con 0x5574052b2bd0 2024-03-24T00:39:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52950/0 (socket says 172.21.15.123:52950) 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c83e00640 1 -- 172.21.15.123:0/2183031372 learned_addr learned my addr 172.21.15.123:0/2183031372 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.190+0000 7f9c88e00640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 0x5574052b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c7b400640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b2bd0 0x5574052c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 -- 172.21.15.123:0/2183031372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b2bd0 msgr2=0x5574052c38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b2bd0 0x5574052c38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 -- 172.21.15.123:0/2183031372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 msgr2=0x5574052b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 0x5574052b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 -- 172.21.15.123:0/2183031372 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574052ae1a0 con 0x5574052b21e0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c88e00640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 0x5574052b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c83e00640 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009900 tx=0x7f9c74042040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e59b5568dbac3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c78c00640 1 -- 172.21.15.123:0/2183031372 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74044040 con 0x5574052b21e0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c78c00640 1 -- 172.21.15.123:0/2183031372 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c7402cc00 con 0x5574052b21e0 2024-03-24T00:39:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.191+0000 7f9c78c00640 1 -- 172.21.15.123:0/2183031372 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74004e10 con 0x5574052b21e0 2024-03-24T00:39:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2183031372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 msgr2=0x5574052b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009900 tx=0x7f9c74042040 comp rx=0 tx=0).stop 2024-03-24T00:39:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2183031372 shutdown_connections 2024-03-24T00:39:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b7d40 0x5574052b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b2bd0 0x5574052c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/2183031372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b21e0 0x5574052b3ef0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2183031372 >> 172.21.15.123:0/2183031372 conn(0x5574052ad750 msgr2=0x5574052bc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2183031372 shutdown_connections 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.192+0000 7f9c8dba7740 1 -- 172.21.15.123:0/2183031372 wait complete. 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 -- start start 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 Processor -- start 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841a9870 con 0x5574052b21e0 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841aab90 con 0x5574052b2bd0 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841a9ed0 con 0x5574052b7d40 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.193+0000 7f9c8dba7740 10 client.0 _ll_register_callbacks cb 0x7f9c7c004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43918/0 (socket says 172.21.15.123:43918) 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:52956/0 (socket says 172.21.15.123:52956) 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 -- 172.21.15.123:0/3534483682 learned_addr learned my addr 172.21.15.123:0/3534483682 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c88e00640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 -- 172.21.15.123:0/3534483682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 msgr2=0x5574052c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 -- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 msgr2=0x5574052c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 -- 172.21.15.123:0/3534483682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574052c2fe0 con 0x5574052b21e0 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c7b400640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:32.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c83e00640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402c8e0 tx=0x7f9c7402ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bba6e5aca7521a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.194+0000 7f9c72000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74044040 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c7402cc00 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c74004250 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c74004450 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c8dba7740 10 client.15423 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557405323f10 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.195+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c7402cc00 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9c74040710 con 0x5574052b21e0 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 10 client.15423 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 20 client.15423 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557405324450 0x557405326880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 4 client.15423 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:39:32.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55740521f0c0 con 0x557405324450 2024-03-24T00:39:32.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.196+0000 7f9c7b400640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557405324450 0x557405326880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.197+0000 7f9c72000640 10 client.15423 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:32.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.197+0000 7f9c7b400640 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557405324450 0x557405326880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1752d63a2191f457 server_cookie=ecbb5b534de7e59a in_seq=0 out_seq=0 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.291+0000 7f9c72000640 1 -- 172.21.15.123:0/3534483682 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55740521f0c0 con 0x557405324450 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.291+0000 7f9c72000640 10 client.15423 handle_command_reply: tid=0 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.291+0000 7f9c8dba7740 1 client.15423 shutdown 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 msgr2=0x5574052c14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402c8e0 tx=0x7f9c7402ca90 comp rx=0 tx=0).stop 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 shutdown_connections 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557405324450 0x557405326880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574052b7d40 0x5574052c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574052b2bd0 0x5574052c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 --2- 172.21.15.123:0/3534483682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5574052b21e0 0x5574052c14f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c72000640 0 client.15423 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 >> 172.21.15.123:0/3534483682 conn(0x5574052ad750 msgr2=0x5574052b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 shutdown_connections 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 1 -- 172.21.15.123:0/3534483682 wait complete. 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.292+0000 7f9c8dba7740 20 client.15423 trim_cache size 0 max 16384 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:39:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:39:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 msgr2=0x7f9c680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 0x7f9c680a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x5574052b3430 tx=0x7f9c6c005850 comp rx=0 tx=0).stop 2024-03-24T00:39:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 msgr2=0x7f9c841a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00eb10 tx=0x7f9c7c00c2d0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 shutdown_connections 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c84103280 0x7f9c841a1100 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c680a2920 0x7f9c680a4de0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c841028c0 0x7f9c8419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 --2- 172.21.15.123:0/307134124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c841016d0 0x7f9c84199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 >> 172.21.15.123:0/307134124 conn(0x7f9c840fcc60 msgr2=0x7f9c840738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 shutdown_connections 2024-03-24T00:39:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:32.293+0000 7f9c8ae00640 1 -- 172.21.15.123:0/307134124 wait complete. 2024-03-24T00:39:32.315 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/vfs.h usability... yes 2024-03-24T00:39:32.372 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/vfs.h presence... yes 2024-03-24T00:39:32.373 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/vfs.h... yes 2024-03-24T00:39:32.474 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/limits.h usability... no 2024-03-24T00:39:32.514 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:32 smithi177 ceph-mon[36034]: pgmap v434: 97 pgs: 97 active+clean; 3.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 267 KiB/s wr, 55 op/s 2024-03-24T00:39:32.549 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/limits.h presence... no 2024-03-24T00:39:32.552 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/limits.h... no 2024-03-24T00:39:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:32 smithi204 ceph-mon[36345]: pgmap v434: 97 pgs: 97 active+clean; 3.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 267 KiB/s wr, 55 op/s 2024-03-24T00:39:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:32 smithi123 ceph-mon[31487]: pgmap v434: 97 pgs: 97 active+clean; 3.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 267 KiB/s wr, 55 op/s 2024-03-24T00:39:32.667 INFO:tasks.workunit.client.0.smithi123.stdout:checking for an ANSI C-conforming const... yes 2024-03-24T00:39:32.783 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-24T00:39:32.787 INFO:tasks.workunit.client.0.smithi123.stdout:checking for function prototypes... yes 2024-03-24T00:39:32.907 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether setvbuf arguments are reversed... no 2024-03-24T00:39:33.064 INFO:tasks.workunit.client.0.smithi123.stdout:checking for system... yes 2024-03-24T00:39:33.234 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gettimeofday... yes 2024-03-24T00:39:33.410 INFO:tasks.workunit.client.0.smithi123.stdout:checking for mkdir... yes 2024-03-24T00:39:33.588 INFO:tasks.workunit.client.0.smithi123.stdout:checking for strerror... yes 2024-03-24T00:39:33.770 INFO:tasks.workunit.client.0.smithi123.stdout:checking for open64... yes 2024-03-24T00:39:33.931 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stat64... yes 2024-03-24T00:39:34.101 INFO:tasks.workunit.client.0.smithi123.stdout:checking for fseeko64... yes 2024-03-24T00:39:34.277 INFO:tasks.workunit.client.0.smithi123.stdout:checking for lrand48_r... yes 2024-03-24T00:39:34.461 INFO:tasks.workunit.client.0.smithi123.stdout:checking for srand48_r... yes 2024-03-24T00:39:34.516 INFO:tasks.workunit.client.0.smithi123.stdout:configure: creating ./config.status 2024-03-24T00:39:34.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:34 smithi204 ceph-mon[36345]: pgmap v435: 97 pgs: 97 active+clean; 3.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 298 KiB/s wr, 86 op/s 2024-03-24T00:39:34.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:34 smithi123 ceph-mon[31487]: pgmap v435: 97 pgs: 97 active+clean; 3.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 298 KiB/s wr, 86 op/s 2024-03-24T00:39:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:34 smithi177 ceph-mon[36034]: pgmap v435: 97 pgs: 97 active+clean; 3.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 298 KiB/s wr, 86 op/s 2024-03-24T00:39:34.832 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: creating Makefile 2024-03-24T00:39:34.883 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: creating config.h 2024-03-24T00:39:34.952 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: executing depfiles commands 2024-03-24T00:39:35.154 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: executing default commands 2024-03-24T00:39:35.244 INFO:tasks.workunit.client.0.smithi123.stderr:+ make 2024-03-24T00:39:35.270 INFO:tasks.workunit.client.0.smithi123.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-24T00:39:35.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-24T00:39:35.281 INFO:tasks.workunit.client.0.smithi123.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-24T00:39:35.281 INFO:tasks.workunit.client.0.smithi123.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-24T00:39:35.282 INFO:tasks.workunit.client.0.smithi123.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-24T00:39:35.282 INFO:tasks.workunit.client.0.smithi123.stderr: any GNU archive site. 2024-03-24T00:39:35.288 INFO:tasks.workunit.client.0.smithi123.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-24T00:39:35.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-24T00:39:35.295 INFO:tasks.workunit.client.0.smithi123.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-24T00:39:35.296 INFO:tasks.workunit.client.0.smithi123.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-24T00:39:35.296 INFO:tasks.workunit.client.0.smithi123.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-24T00:39:35.296 INFO:tasks.workunit.client.0.smithi123.stderr: Grab them from any GNU archive site. 2024-03-24T00:39:35.399 INFO:tasks.workunit.client.0.smithi123.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-24T00:39:35.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:39:35.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:39:35.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.822+0000 7f10f6e00640 1 -- 172.21.15.123:0/3321569704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0076060 msgr2=0x7f10f0076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.822+0000 7f10f6e00640 1 --2- 172.21.15.123:0/3321569704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0076060 0x7f10f0076480 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009900 tx=0x7f10e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:35.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 -- 172.21.15.123:0/3321569704 shutdown_connections 2024-03-24T00:39:35.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 --2- 172.21.15.123:0/3321569704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0077c40 0x7f10f00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 --2- 172.21.15.123:0/3321569704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f0077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 --2- 172.21.15.123:0/3321569704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0076060 0x7f10f0076480 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 -- 172.21.15.123:0/3321569704 >> 172.21.15.123:0/3321569704 conn(0x7f10f00fee50 msgr2=0x7f10f0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 -- 172.21.15.123:0/3321569704 shutdown_connections 2024-03-24T00:39:35.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.823+0000 7f10f6e00640 1 -- 172.21.15.123:0/3321569704 wait complete. 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 Processor -- start 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 -- start start 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0076060 0x7f10f019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 0x7f10f01a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01139b0 con 0x7f10f0076060 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0113830 con 0x7f10f0077c40 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.824+0000 7f10f6e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0113b30 con 0x7f10f00772a0 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10efe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0076060 0x7f10f019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50472/0 (socket says 172.21.15.123:50472) 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 -- 172.21.15.123:0/2351374103 learned_addr learned my addr 172.21.15.123:0/2351374103 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:35.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 0x7f10f01a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 -- 172.21.15.123:0/2351374103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 msgr2=0x7f10f01a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 0x7f10f01a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 -- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0076060 msgr2=0x7f10f019e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0076060 0x7f10f019e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 -- 172.21.15.123:0/2351374103 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f01a5ce0 con 0x7f10f00772a0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 0x7f10f01a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.825+0000 7f10e7400640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e9f0 tx=0x7f10d800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.826+0000 7f10ed600640 1 -- 172.21.15.123:0/2351374103 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d800cd80 con 0x7f10f00772a0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.826+0000 7f10ed600640 1 -- 172.21.15.123:0/2351374103 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10d8005500 con 0x7f10f00772a0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.826+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01a5fd0 con 0x7f10f00772a0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.826+0000 7f10ed600640 1 -- 172.21.15.123:0/2351374103 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8010690 con 0x7f10f00772a0 2024-03-24T00:39:35.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.826+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f01a6490 con 0x7f10f00772a0 2024-03-24T00:39:35.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.829+0000 7f10f9bb2740 1 Processor -- start 2024-03-24T00:39:35.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.829+0000 7f10f9bb2740 1 -- start start 2024-03-24T00:39:35.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.829+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373de80 0x56226373e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.829+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5622637439e0 0x562263745e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622636a3250 con 0x56226373de80 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226364ba30 con 0x56226373e870 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226369f2f0 con 0x5622637439e0 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10ed600640 1 -- 172.21.15.123:0/2351374103 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f10d80046c0 con 0x7f10f00772a0 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10f4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373de80 0x56226373e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54632/0 (socket says 172.21.15.123:54632) 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 -- 172.21.15.123:0/2710517054 learned_addr learned my addr 172.21.15.123:0/2710517054 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10e7400640 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5622637439e0 0x562263745e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 -- 172.21.15.123:0/2710517054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5622637439e0 msgr2=0x562263745e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5622637439e0 0x562263745e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 -- 172.21.15.123:0/2710517054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373de80 msgr2=0x56226373e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373de80 0x56226373e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.830+0000 7f10efe00640 1 -- 172.21.15.123:0/2710517054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562263636690 con 0x56226373e870 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.831+0000 7f10efe00640 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10e002c8e0 tx=0x7f10e00095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=126a3869ad929f89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.831+0000 7f10e5600640 1 -- 172.21.15.123:0/2710517054 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0045070 con 0x56226373e870 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.831+0000 7f10e5600640 1 -- 172.21.15.123:0/2710517054 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e0004040 con 0x56226373e870 2024-03-24T00:39:35.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.831+0000 7f10e5600640 1 -- 172.21.15.123:0/2710517054 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0004ef0 con 0x56226373e870 2024-03-24T00:39:35.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2710517054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 msgr2=0x562263743370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10e002c8e0 tx=0x7f10e00095e0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2710517054 shutdown_connections 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5622637439e0 0x562263745e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373e870 0x562263743370 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2710517054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373de80 0x56226373e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2710517054 >> 172.21.15.123:0/2710517054 conn(0x562263706200 msgr2=0x5622637397f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2710517054 shutdown_connections 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2710517054 wait complete. 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10ed600640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 0x7f10d00a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.832+0000 7f10f9bb2740 1 -- start start 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10ed600640 1 -- 172.21.15.123:0/2351374103 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f10d80cee20 con 0x7f10f00772a0 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 0x7f10d00a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 Processor -- start 2024-03-24T00:39:35.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x5622637482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562263746930 con 0x56226373e870 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226365ee80 con 0x5622637439e0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562263746ab0 con 0x56226373de80 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f9bb2740 10 client.0 _ll_register_callbacks cb 0x562263671130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.833+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 0x7f10d00a4c50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x56226373f0d0 tx=0x7f10e80096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54644/0 (socket says 172.21.15.123:54644) 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 -- 172.21.15.123:0/840936632 learned_addr learned my addr 172.21.15.123:0/840936632 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10efe00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10e7400640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x5622637482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 -- 172.21.15.123:0/840936632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 msgr2=0x562263747d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 -- 172.21.15.123:0/840936632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 msgr2=0x5622637482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x5622637482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 -- 172.21.15.123:0/840936632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622634817d0 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10e7400640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x5622637482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10f4e00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f10e8010c40 tx=0x7f10e8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=143afd3e08a44453 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.834+0000 7f10de000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e801fd40 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f10e8004b20 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10e0004040 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10e8004d60 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10f9bb2740 10 client.25541 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5622633b73f0 con 0x5622637439e0 2024-03-24T00:39:35.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e0004040 con 0x5622637439e0 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f10e8026070 con 0x5622637439e0 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10f9bb2740 10 client.25541 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.835+0000 7f10f9bb2740 10 client.25541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10f9bb2740 20 client.25541 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562263753c00 0x562263756040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10f9bb2740 4 client.25541 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622636b0790 con 0x562263753c00 2024-03-24T00:39:35.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10efe00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562263753c00 0x562263756040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10efe00640 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562263753c00 0x562263756040 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a8f9b018a1b24 server_cookie=abaf7279684e00eb in_seq=0 out_seq=0 2024-03-24T00:39:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.836+0000 7f10de000640 10 client.25541 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.837+0000 7f10de000640 1 -- 172.21.15.123:0/840936632 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622636b0790 con 0x562263753c00 2024-03-24T00:39:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.837+0000 7f10de000640 10 client.25541 handle_command_reply: tid=0 2024-03-24T00:39:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.837+0000 7f10f9bb2740 1 client.25541 shutdown 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.837+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 msgr2=0x56226374f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.837+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f10e8010c40 tx=0x7f10e8004ab0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 shutdown_connections 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x56226374f310 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x5622637482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562263753c00 0x562263756040 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/840936632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10de000640 0 client.25541 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 >> 172.21.15.123:0/840936632 conn(0x562263706200 msgr2=0x562263738a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 shutdown_connections 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 1 -- 172.21.15.123:0/840936632 wait complete. 2024-03-24T00:39:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.838+0000 7f10f9bb2740 20 client.25541 trim_cache size 0 max 16384 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 Processor -- start 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 -- start start 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x56226374f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01ad0c0 con 0x56226373e870 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01af680 con 0x5622637439e0 2024-03-24T00:39:35.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.863+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01ad6c0 con 0x56226373de80 2024-03-24T00:39:35.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54658/0 (socket says 172.21.15.123:54658) 2024-03-24T00:39:35.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10e7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x56226374f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 -- 172.21.15.123:0/2650712417 learned_addr learned my addr 172.21.15.123:0/2650712417 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10efe00640 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 -- 172.21.15.123:0/2650712417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 msgr2=0x562263747d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 -- 172.21.15.123:0/2650712417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 msgr2=0x56226374f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x56226374f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 -- 172.21.15.123:0/2650712417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622633770f0 con 0x5622637439e0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.864+0000 7f10f4e00640 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006010 tx=0x7f10e8010920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a20e8f1a4cbeb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10e4c00640 1 -- 172.21.15.123:0/2650712417 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8004ba0 con 0x5622637439e0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10e4c00640 1 -- 172.21.15.123:0/2650712417 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e8036020 con 0x5622637439e0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10e4c00640 1 -- 172.21.15.123:0/2650712417 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e80042a0 con 0x5622637439e0 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2650712417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 msgr2=0x562263738910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006010 tx=0x7f10e8010920 comp rx=0 tx=0).stop 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2650712417 shutdown_connections 2024-03-24T00:39:35.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622637439e0 0x562263738910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56226373e870 0x56226374f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/2650712417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373de80 0x562263747d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.865+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2650712417 >> 172.21.15.123:0/2650712417 conn(0x562263706200 msgr2=0x562263738200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.866+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2650712417 shutdown_connections 2024-03-24T00:39:35.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.866+0000 7f10f9bb2740 1 -- 172.21.15.123:0/2650712417 wait complete. 2024-03-24T00:39:35.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.866+0000 7f10f9bb2740 1 -- start start 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.866+0000 7f10f9bb2740 1 Processor -- start 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 0x56226374d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 0x56226374e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01add00 con 0x5622637439e0 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01af020 con 0x56226373de80 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01ae360 con 0x56226373e870 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f9bb2740 10 client.0 _ll_register_callbacks cb 0x7f10e8010ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10e7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 0x56226374d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54668/0 (socket says 172.21.15.123:54668) 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 -- 172.21.15.123:0/3250623298 learned_addr learned my addr 172.21.15.123:0/3250623298 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f4e00640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 0x56226374e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 -- 172.21.15.123:0/3250623298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 msgr2=0x56226374d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 0x56226374d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 -- 172.21.15.123:0/3250623298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 msgr2=0x56226374e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 0x56226374e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 -- 172.21.15.123:0/3250623298 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56226374ed40 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10f4e00640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 0x56226374e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10e7400640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 0x56226374d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10efe00640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005ab0 tx=0x7f10e0005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ff41c3f0720ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.867+0000 7f10de000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0056030 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f10e002c980 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10e8036020 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10e0045040 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 10 client.25547 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5622637afd90 con 0x56226373de80 2024-03-24T00:39:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e8036020 con 0x56226373de80 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f10e0005060 con 0x56226373de80 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 10 client.25547 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 10 client.25547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 20 client.25547 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5622637b02c0 0x5622637b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 4 client.25547 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.868+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5622636aaae0 con 0x5622637b02c0 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10e7400640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5622637b02c0 0x5622637b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10e7400640 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5622637b02c0 0x5622637b26e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aab33d57104f1e0 server_cookie=1214d0ec60edbe10 in_seq=0 out_seq=0 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10de000640 10 client.25547 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10de000640 1 -- 172.21.15.123:0/3250623298 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5622636aaae0 con 0x5622637b02c0 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10de000640 10 client.25547 handle_command_reply: tid=0 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 client.25547 shutdown 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 msgr2=0x56226374d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005ab0 tx=0x7f10e0005e50 comp rx=0 tx=0).stop 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 shutdown_connections 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5622637439e0 0x56226374e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5622637b02c0 0x5622637b26e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56226373e870 0x56226374d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 --2- 172.21.15.123:0/3250623298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56226373de80 0x56226374d200 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10de000640 0 client.25547 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.869+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 >> 172.21.15.123:0/3250623298 conn(0x562263706200 msgr2=0x562263738a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 shutdown_connections 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f9bb2740 1 -- 172.21.15.123:0/3250623298 wait complete. 2024-03-24T00:39:35.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f9bb2740 20 client.25547 trim_cache size 0 max 16384 2024-03-24T00:39:35.875 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:35.875 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:39:35.875 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:39:35.875 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:35.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 msgr2=0x7f10d00a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 0x7f10d00a4c50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x56226373f0d0 tx=0x7f10e80096a0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 msgr2=0x7f10f019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e9f0 tx=0x7f10d800edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 shutdown_connections 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0077c40 0x7f10f01a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10f00772a0 0x7f10f019e600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f10d00a2790 0x7f10d00a4c50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 --2- 172.21.15.123:0/2351374103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10f0076060 0x7f10f019e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.870+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 >> 172.21.15.123:0/2351374103 conn(0x7f10f00fee50 msgr2=0x7f10f0100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.871+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 shutdown_connections 2024-03-24T00:39:35.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:35.871+0000 7f10f6e00640 1 -- 172.21.15.123:0/2351374103 wait complete. 2024-03-24T00:39:35.888 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:39:35.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:35.929 INFO:tasks.workunit.client.0.smithi123.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-24T00:39:35.929 INFO:tasks.workunit.client.0.smithi123.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-24T00:39:35.929 INFO:tasks.workunit.client.0.smithi123.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-24T00:39:35.929 INFO:tasks.workunit.client.0.smithi123.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 -- 172.21.15.123:0/3311204708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360104ad0 msgr2=0x7fb360104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 --2- 172.21.15.123:0/3311204708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360104ad0 0x7fb360104f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009900 tx=0x7fb34c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 -- 172.21.15.123:0/3311204708 shutdown_connections 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 --2- 172.21.15.123:0/3311204708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360105490 0x7fb36010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 --2- 172.21.15.123:0/3311204708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360104ad0 0x7fb360104f50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 --2- 172.21.15.123:0/3311204708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 0x7fb360103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 -- 172.21.15.123:0/3311204708 >> 172.21.15.123:0/3311204708 conn(0x7fb3600fee90 msgr2=0x7fb3601012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 -- 172.21.15.123:0/3311204708 shutdown_connections 2024-03-24T00:39:36.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.077+0000 7fb365a00640 1 -- 172.21.15.123:0/3311204708 wait complete. 2024-03-24T00:39:36.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.078+0000 7fb365a00640 1 Processor -- start 2024-03-24T00:39:36.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.078+0000 7fb365a00640 1 -- start start 2024-03-24T00:39:36.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.078+0000 7fb365a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 0x7fb360199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.078+0000 7fb365a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360104ad0 0x7fb36019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.078+0000 7fb365a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb365a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36010f560 con 0x7fb3601038e0 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb365a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36010f3e0 con 0x7fb360105490 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb365a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36010f6e0 con 0x7fb360104ad0 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 0x7fb360199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54686/0 (socket says 172.21.15.123:54686) 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 -- 172.21.15.123:0/705635507 learned_addr learned my addr 172.21.15.123:0/705635507 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35e000640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360104ad0 0x7fb36019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 -- 172.21.15.123:0/705635507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360104ad0 msgr2=0x7fb36019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360104ad0 0x7fb36019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 -- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 msgr2=0x7fb360199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 0x7fb360199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 -- 172.21.15.123:0/705635507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3601a17f0 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.079+0000 7fb35f400640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb35800eb10 tx=0x7fb35800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.080+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358004b20 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.080+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb358004cc0 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.080+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358010760 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.080+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3601a1ae0 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.080+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3601a1fb0 con 0x7fb360105490 2024-03-24T00:39:36.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.081+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36010ac00 con 0x7fb360105490 2024-03-24T00:39:36.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.083+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb358005020 con 0x7fb360105490 2024-03-24T00:39:36.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.084+0000 7fb357400640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 0x7fb3440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.084+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb3580cf1b0 con 0x7fb360105490 2024-03-24T00:39:36.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.084+0000 7fb35ea00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 0x7fb3440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.085+0000 7fb35ea00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 0x7fb3440a4d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb360104930 tx=0x7fb350005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.088+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb35809c200 con 0x7fb360105490 2024-03-24T00:39:36.106 INFO:tasks.workunit.client.0.smithi123.stdout:/bin/sh ./config.status --recheck 2024-03-24T00:39:36.262 INFO:tasks.workunit.client.0.smithi123.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-24T00:39:36.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.335+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb36002d170 con 0x7fb360105490 2024-03-24T00:39:36.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.336+0000 7fb357400640 1 -- 172.21.15.123:0/705635507 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fb35809bb00 con 0x7fb360105490 2024-03-24T00:39:36.336 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:39:36.336 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:39:36.337 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 msgr2=0x7fb3440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 0x7fb3440a4d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb360104930 tx=0x7fb350005f90 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 msgr2=0x7fb3601a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb35800eb10 tx=0x7fb35800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 shutdown_connections 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb360105490 0x7fb3601a1180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb360104ad0 0x7fb36019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb3440a2880 0x7fb3440a4d40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 --2- 172.21.15.123:0/705635507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3601038e0 0x7fb360199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 >> 172.21.15.123:0/705635507 conn(0x7fb3600fee90 msgr2=0x7fb360107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 shutdown_connections 2024-03-24T00:39:36.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.339+0000 7fb365a00640 1 -- 172.21.15.123:0/705635507 wait complete. 2024-03-24T00:39:36.357 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:39:36.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:39:36.523 INFO:tasks.workunit.client.0.smithi123.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T00:39:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:36 smithi204 ceph-mon[36345]: pgmap v436: 97 pgs: 97 active+clean; 4.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 422 KiB/s wr, 150 op/s 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 -- 172.21.15.123:0/571051880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8104ab0 msgr2=0x7f35b8104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 --2- 172.21.15.123:0/571051880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8104ab0 0x7f35b8104f30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009900 tx=0x7f35a002bc50 comp rx=0 tx=0).stop 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 -- 172.21.15.123:0/571051880 shutdown_connections 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 --2- 172.21.15.123:0/571051880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b8105470 0x7f35b810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 --2- 172.21.15.123:0/571051880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8104ab0 0x7f35b8104f30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 --2- 172.21.15.123:0/571051880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b81038c0 0x7f35b8103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 -- 172.21.15.123:0/571051880 >> 172.21.15.123:0/571051880 conn(0x7f35b80fee50 msgr2=0x7f35b8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 -- 172.21.15.123:0/571051880 shutdown_connections 2024-03-24T00:39:36.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.602+0000 7f35bd000640 1 -- 172.21.15.123:0/571051880 wait complete. 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.603+0000 7f35bd000640 1 Processor -- start 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.603+0000 7f35bd000640 1 -- start start 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.603+0000 7f35bd000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b81038c0 0x7f35b8199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.603+0000 7f35bd000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 0x7f35b819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.603+0000 7f35bd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35bd000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b810f570 con 0x7f35b81038c0 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35bd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b810f3f0 con 0x7f35b8105470 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35bd000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b810f6f0 con 0x7f35b8104ab0 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b81038c0 0x7f35b8199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54700/0 (socket says 172.21.15.123:54700) 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 -- 172.21.15.123:0/427248741 learned_addr learned my addr 172.21.15.123:0/427248741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35afe00640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 0x7f35b819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 -- 172.21.15.123:0/427248741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 msgr2=0x7f35b819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 0x7f35b819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 -- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b81038c0 msgr2=0x7f35b8199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b81038c0 0x7f35b8199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 -- 172.21.15.123:0/427248741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b81a17d0 con 0x7f35b8105470 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35afe00640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 0x7f35b819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.604+0000 7f35b6800640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35b000eb10 tx=0x7f35b000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.605+0000 7f35aea00640 1 -- 172.21.15.123:0/427248741 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0004b20 con 0x7f35b8105470 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.605+0000 7f35aea00640 1 -- 172.21.15.123:0/427248741 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35b0004cc0 con 0x7f35b8105470 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.605+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b81a1ac0 con 0x7f35b8105470 2024-03-24T00:39:36.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.605+0000 7f35aea00640 1 -- 172.21.15.123:0/427248741 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0010760 con 0x7f35b8105470 2024-03-24T00:39:36.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.605+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b81a1f90 con 0x7f35b8105470 2024-03-24T00:39:36.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:36 smithi123 ceph-mon[31487]: pgmap v436: 97 pgs: 97 active+clean; 4.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 422 KiB/s wr, 150 op/s 2024-03-24T00:39:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.608+0000 7f35aea00640 1 -- 172.21.15.123:0/427248741 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f35b0005020 con 0x7f35b8105470 2024-03-24T00:39:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.608+0000 7f35bfdf3740 1 Processor -- start 2024-03-24T00:39:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.608+0000 7f35bfdf3740 1 -- start start 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7eac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff74f2c0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff6f7c90 con 0x5569ff7ec3a0 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff74b360 con 0x5569ff7ea810 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35b5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54702/0 (socket says 172.21.15.123:54702) 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35afe00640 1 -- 172.21.15.123:0/710863468 learned_addr learned my addr 172.21.15.123:0/710863468 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35b5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53536/0 (socket says 172.21.15.123:53536) 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.609+0000 7f35afe00640 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7eac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 -- 172.21.15.123:0/710863468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 msgr2=0x5569ff7eac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7eac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 -- 172.21.15.123:0/710863468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 msgr2=0x5569ff7f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 -- 172.21.15.123:0/710863468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569ff71c080 con 0x5569ff7eb9e0 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35afe00640 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35aea00640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 0x7f35980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35b5e00640 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f35b8104910 tx=0x7f35a800bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a083a57743bc6b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35aea00640 1 -- 172.21.15.123:0/427248741 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f35b00cf060 con 0x7f35b8105470 2024-03-24T00:39:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35afe00640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 0x7f35980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35a7400640 1 -- 172.21.15.123:0/710863468 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8003dd0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35a7400640 1 -- 172.21.15.123:0/710863468 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35a8015070 con 0x5569ff7eb9e0 2024-03-24T00:39:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.610+0000 7f35a7400640 1 -- 172.21.15.123:0/710863468 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a80116a0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 -- 172.21.15.123:0/710863468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 msgr2=0x5569ff7ebe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35afe00640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 0x7f35980a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5569ff7eb8e0 tx=0x7f35a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f35b8104910 tx=0x7f35a800bbe0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 -- 172.21.15.123:0/710863468 shutdown_connections 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7ec3a0 0x5569ff7f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7eb9e0 0x5569ff7ebe60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/710863468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 -- 172.21.15.123:0/710863468 >> 172.21.15.123:0/710863468 conn(0x5569ff7e5c00 msgr2=0x5569ff7e6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 -- 172.21.15.123:0/710863468 shutdown_connections 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.611+0000 7f35bfdf3740 1 -- 172.21.15.123:0/710863468 wait complete. 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 -- start start 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 Processor -- start 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff7f33b0 con 0x5569ff7ea810 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff70b0d0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ff7f3530 con 0x5569ff7ec3a0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.612+0000 7f35bfdf3740 10 client.0 _ll_register_callbacks cb 0x5569ff70fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54704/0 (socket says 172.21.15.123:54704) 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 -- 172.21.15.123:0/3060791804 learned_addr learned my addr 172.21.15.123:0/3060791804 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35b5e00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35b6800640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 -- 172.21.15.123:0/3060791804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 msgr2=0x5569ff7fc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 -- 172.21.15.123:0/3060791804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 msgr2=0x5569ff7f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 -- 172.21.15.123:0/3060791804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569ff4633f0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f35afe00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f35a002c260 tx=0x7f35a0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=379aa15533421f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.613+0000 7f359fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a004f070 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f35a0042640 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35a8015070 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35a005b050 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 10 client.25565 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5569ff52d7d0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35a8015070 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f35a00349d0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 10 client.25565 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 20 client.25565 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.614+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff800470 0x5569ff8028e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.615+0000 7f35bfdf3740 4 client.25565 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.615+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569ff75c800 con 0x5569ff800470 2024-03-24T00:39:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.615+0000 7f35b5e00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff800470 0x5569ff8028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.615+0000 7f359fe00640 10 client.25565 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.615+0000 7f35b5e00640 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff800470 0x5569ff8028e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39cb6ad04c76505a server_cookie=8aea9837b3b6c094 in_seq=0 out_seq=0 2024-03-24T00:39:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.616+0000 7f359fe00640 1 -- 172.21.15.123:0/3060791804 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5569ff75c800 con 0x5569ff800470 2024-03-24T00:39:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.616+0000 7f359fe00640 10 client.25565 handle_command_reply: tid=0 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.616+0000 7f35bfdf3740 1 client.25565 shutdown 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.616+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 msgr2=0x5569ff7f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.616+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f35a002c260 tx=0x7f35a0009260 comp rx=0 tx=0).stop 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 shutdown_connections 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff800470 0x5569ff8028e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f4ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f359fe00640 0 client.25565 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3060791804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 >> 172.21.15.123:0/3060791804 conn(0x5569ff7e5c00 msgr2=0x5569ff7e4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 shutdown_connections 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3060791804 wait complete. 2024-03-24T00:39:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.617+0000 7f35bfdf3740 20 client.25565 trim_cache size 0 max 16384 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 Processor -- start 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 -- start start 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ec3a0 0x5569ff7ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81a8c50 con 0x5569ff7ec3a0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81ab210 con 0x5569ff7eb9e0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.630+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81a9250 con 0x5569ff7ea810 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35b6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ec3a0 0x5569ff7ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54706/0 (socket says 172.21.15.123:54706) 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 -- 172.21.15.123:0/1931140056 learned_addr learned my addr 172.21.15.123:0/1931140056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35b5e00640 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 -- 172.21.15.123:0/1931140056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 msgr2=0x5569ff7f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 -- 172.21.15.123:0/1931140056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ec3a0 msgr2=0x5569ff7ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ec3a0 0x5569ff7ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 -- 172.21.15.123:0/1931140056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569ff7e6650 con 0x5569ff7eb9e0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.631+0000 7f35afe00640 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f35a0034d70 tx=0x7f35a0004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a23801b00d104076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35a6a00640 1 -- 172.21.15.123:0/1931140056 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0058020 con 0x5569ff7eb9e0 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35a6a00640 1 -- 172.21.15.123:0/1931140056 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35a003fc90 con 0x5569ff7eb9e0 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35a6a00640 1 -- 172.21.15.123:0/1931140056 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a004f040 con 0x5569ff7eb9e0 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 -- 172.21.15.123:0/1931140056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 msgr2=0x5569ff7fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f35a0034d70 tx=0x7f35a0004480 comp rx=0 tx=0).stop 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 -- 172.21.15.123:0/1931140056 shutdown_connections 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ec3a0 0x5569ff7ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7fbff0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/1931140056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ea810 0x5569ff7f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 -- 172.21.15.123:0/1931140056 >> 172.21.15.123:0/1931140056 conn(0x5569ff7e5c00 msgr2=0x5569ff7e96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 -- 172.21.15.123:0/1931140056 shutdown_connections 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.632+0000 7f35bfdf3740 1 -- 172.21.15.123:0/1931140056 wait complete. 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 -- start start 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 Processor -- start 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81a9890 con 0x5569ff7ea810 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81aabb0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81a9ef0 con 0x5569ff7ec3a0 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.633+0000 7f35bfdf3740 10 client.0 _ll_register_callbacks cb 0x7f35a00040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35b5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54718/0 (socket says 172.21.15.123:54718) 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 -- 172.21.15.123:0/3520026354 learned_addr learned my addr 172.21.15.123:0/3520026354 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35b6800640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 -- 172.21.15.123:0/3520026354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 msgr2=0x5569ff7fb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 -- 172.21.15.123:0/3520026354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 msgr2=0x5569ff7f7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 -- 172.21.15.123:0/3520026354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569ff7fb8d0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35b6800640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35afe00640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009850 tx=0x7f35a0004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8af18ce044ebb2d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f359fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0058030 con 0x5569ff7eb9e0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f35a003f430 con 0x5569ff7eb9e0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.634+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35a003fc90 con 0x5569ff7eb9e0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35a004f070 con 0x5569ff7eb9e0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 10 client.25571 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5569ff85c3f0 con 0x5569ff7eb9e0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35a003fc90 con 0x5569ff7eb9e0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f35a0066030 con 0x5569ff7eb9e0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 10 client.25571 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 10 client.25571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 20 client.25571 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff85cd50 0x5569ff85f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 4 client.25571 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5569ff756b50 con 0x5569ff85cd50 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.635+0000 7f35b5e00640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff85cd50 0x5569ff85f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35b5e00640 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff85cd50 0x5569ff85f140 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9438ef8d7b0d0afc server_cookie=fa46f724a8ff328e in_seq=0 out_seq=0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f359fe00640 10 client.25571 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f359fe00640 1 -- 172.21.15.123:0/3520026354 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5569ff756b50 con 0x5569ff85cd50 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f359fe00640 10 client.25571 handle_command_reply: tid=0 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 client.25571 shutdown 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 msgr2=0x5569ff7f8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009850 tx=0x7f35a0004610 comp rx=0 tx=0).stop 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 shutdown_connections 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5569ff85cd50 0x5569ff85f140 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ff7ec3a0 0x5569ff7fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5569ff7eb9e0 0x5569ff7f8310 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 --2- 172.21.15.123:0/3520026354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5569ff7ea810 0x5569ff7f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f359fe00640 0 client.25571 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 >> 172.21.15.123:0/3520026354 conn(0x5569ff7e5c00 msgr2=0x5569ff7e96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 shutdown_connections 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.636+0000 7f35bfdf3740 1 -- 172.21.15.123:0/3520026354 wait complete. 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bfdf3740 20 client.25571 trim_cache size 0 max 16384 2024-03-24T00:39:36.642 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:39:36.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 msgr2=0x7f35980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 0x7f35980a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5569ff7eb8e0 tx=0x7f35a0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 msgr2=0x7f35b81a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f35b000eb10 tx=0x7f35b000c2d0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 shutdown_connections 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b8105470 0x7f35b81a1160 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35b8104ab0 0x7f35b819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f35980a2800 0x7f35980a4cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 --2- 172.21.15.123:0/427248741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35b81038c0 0x7f35b8199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 >> 172.21.15.123:0/427248741 conn(0x7f35b80fee50 msgr2=0x7f35b8107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 shutdown_connections 2024-03-24T00:39:36.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:36.637+0000 7f35bd000640 1 -- 172.21.15.123:0/427248741 wait complete. 2024-03-24T00:39:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:36 smithi177 ceph-mon[36034]: pgmap v436: 97 pgs: 97 active+clean; 4.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 422 KiB/s wr, 150 op/s 2024-03-24T00:39:37.550 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether build environment is sane... yes 2024-03-24T00:39:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:37 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/705635507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:37.572 INFO:tasks.workunit.client.0.smithi123.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-24T00:39:37.572 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gawk... gawk 2024-03-24T00:39:37.604 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether make sets $(MAKE)... yes 2024-03-24T00:39:37.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:37 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/705635507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:37.654 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:39:37.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:37.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:39:37.700 INFO:tasks.workunit.client.0.smithi123.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-24T00:39:37.700 INFO:tasks.workunit.client.0.smithi123.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-24T00:39:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:37 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/705635507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:39:37.717 INFO:tasks.workunit.client.0.smithi123.stdout:checking for style of include used by make... GNU 2024-03-24T00:39:37.729 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc... gcc 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.858+0000 7f0347200640 1 -- 172.21.15.123:0/793251020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0348103150 msgr2=0x7f03481099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.858+0000 7f0347200640 1 --2- 172.21.15.123:0/793251020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0348103150 0x7f03481099e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0340009ca0 tx=0x7f034002c100 comp rx=0 tx=0).stop 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 -- 172.21.15.123:0/793251020 shutdown_connections 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 --2- 172.21.15.123:0/793251020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0348103150 0x7f03481099e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 --2- 172.21.15.123:0/793251020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348102790 0x7f0348102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 --2- 172.21.15.123:0/793251020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03481015a0 0x7f03481019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 -- 172.21.15.123:0/793251020 >> 172.21.15.123:0/793251020 conn(0x7f03480fcb10 msgr2=0x7f03480fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 -- 172.21.15.123:0/793251020 shutdown_connections 2024-03-24T00:39:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.859+0000 7f0347200640 1 -- 172.21.15.123:0/793251020 wait complete. 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 Processor -- start 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 -- start start 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 0x7f0348075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 0x7f0348073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034810d230 con 0x7f0348103150 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034810d0b0 con 0x7f0348102790 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.860+0000 7f0347200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034810d3b0 con 0x7f03481015a0 2024-03-24T00:39:37.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54746/0 (socket says 172.21.15.123:54746) 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 -- 172.21.15.123:0/2608245411 learned_addr learned my addr 172.21.15.123:0/2608245411 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f0346800640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 0x7f0348073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f0345e00640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 0x7f0348075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 -- 172.21.15.123:0/2608245411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 msgr2=0x7f0348075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 0x7f0348075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 -- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 msgr2=0x7f0348073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 0x7f0348073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 -- 172.21.15.123:0/2608245411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0348074860 con 0x7f0348102790 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f0345e00640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 0x7f0348075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f0346800640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 0x7f0348073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.861+0000 7f033d400640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f033000e9f0 tx=0x7f033000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.862+0000 7f033f400640 1 -- 172.21.15.123:0/2608245411 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033000cd80 con 0x7f0348102790 2024-03-24T00:39:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.862+0000 7f033f400640 1 -- 172.21.15.123:0/2608245411 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0330010910 con 0x7f0348102790 2024-03-24T00:39:37.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.862+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03481a1970 con 0x7f0348102790 2024-03-24T00:39:37.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.862+0000 7f033f400640 1 -- 172.21.15.123:0/2608245411 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330018990 con 0x7f0348102790 2024-03-24T00:39:37.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.862+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03481a1ec0 con 0x7f0348102790 2024-03-24T00:39:37.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.866+0000 7f034fa06740 1 Processor -- start 2024-03-24T00:39:37.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 -- start start 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f033f400640 1 -- 172.21.15.123:0/2608245411 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0330010ac0 con 0x7f0348102790 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a55511eb0 con 0x560a555b1e10 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a554baa60 con 0x560a555ac2b0 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f034fa06740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5550df50 con 0x560a555acca0 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f0346800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f0346800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54756/0 (socket says 172.21.15.123:54756) 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f0346800640 1 -- 172.21.15.123:0/2781207177 learned_addr learned my addr 172.21.15.123:0/2781207177 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f033d400640 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.867+0000 7f0345e00640 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 -- 172.21.15.123:0/2781207177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 msgr2=0x560a555b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 -- 172.21.15.123:0/2781207177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 msgr2=0x560a555b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 -- 172.21.15.123:0/2781207177 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a552ee7e0 con 0x560a555ac2b0 2024-03-24T00:39:37.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0346800640 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0340009ca0 tx=0x7f0340009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2a81282262255d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0337400640 1 -- 172.21.15.123:0/2781207177 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340043020 con 0x560a555ac2b0 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0337400640 1 -- 172.21.15.123:0/2781207177 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0340003970 con 0x560a555ac2b0 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.868+0000 7f0337400640 1 -- 172.21.15.123:0/2781207177 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340039040 con 0x560a555ac2b0 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 -- 172.21.15.123:0/2781207177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 msgr2=0x560a555ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0340009ca0 tx=0x7f0340009800 comp rx=0 tx=0).stop 2024-03-24T00:39:37.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f033f400640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 0x7f032c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 -- 172.21.15.123:0/2781207177 shutdown_connections 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 --2- 172.21.15.123:0/2781207177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555ac6d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 -- 172.21.15.123:0/2781207177 >> 172.21.15.123:0/2781207177 conn(0x560a555a7820 msgr2=0x560a555a7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f033f400640 1 -- 172.21.15.123:0/2608245411 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f03300ceef0 con 0x7f0348102790 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f0346800640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 0x7f032c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 -- 172.21.15.123:0/2781207177 shutdown_connections 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.869+0000 7f034fa06740 1 -- 172.21.15.123:0/2781207177 wait complete. 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.870+0000 7f034fa06740 1 -- start start 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.870+0000 7f0346800640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 0x7f032c0a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0340032420 tx=0x7f034002c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.870+0000 7f034fa06740 1 Processor -- start 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555bd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a555b4d60 con 0x560a555b1e10 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a554cdeb0 con 0x560a555ac2b0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a555b4ee0 con 0x560a555acca0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f034fa06740 10 client.0 _ll_register_callbacks cb 0x560a554c2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54768/0 (socket says 172.21.15.123:54768) 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 -- 172.21.15.123:0/4236280193 learned_addr learned my addr 172.21.15.123:0/4236280193 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0346800640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555bd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f033d400640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 -- 172.21.15.123:0/4236280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 msgr2=0x560a555b6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 -- 172.21.15.123:0/4236280193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 msgr2=0x560a555bd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555bd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0345e00640 1 -- 172.21.15.123:0/4236280193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a5534ef70 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f033d400640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.871+0000 7f0346800640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555bd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f0345e00640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0338012700 tx=0x7f0338012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccc742622894b8de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338004320 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f03380119a0 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0340003970 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03380044c0 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f034fa06740 10 client.25583 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560a551e6170 con 0x560a555ac2b0 2024-03-24T00:39:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0340003970 con 0x560a555ac2b0 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.872+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f033800f440 con 0x560a555ac2b0 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 10 client.25583 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 20 client.25583 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a555c22b0 0x560a555c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 4 client.25583 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a5551f3f0 con 0x560a555c22b0 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f033d400640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a555c22b0 0x560a555c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f032be00640 10 client.25583 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.873+0000 7f033d400640 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a555c22b0 0x560a555c4750 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1fc057ea9c6a07 server_cookie=70b5edd1ad4aaeea in_seq=0 out_seq=0 2024-03-24T00:39:37.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f032be00640 1 -- 172.21.15.123:0/4236280193 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560a5551f3f0 con 0x560a555c22b0 2024-03-24T00:39:37.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f032be00640 10 client.25583 handle_command_reply: tid=0 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f034fa06740 1 client.25583 shutdown 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 msgr2=0x560a555b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0338012700 tx=0x7f0338012bc0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 shutdown_connections 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.874+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555b1e10 0x560a555bd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a555c22b0 0x560a555c4750 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 --2- 172.21.15.123:0/4236280193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555ac2b0 0x560a555b6420 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 >> 172.21.15.123:0/4236280193 conn(0x560a555a7820 msgr2=0x560a555a6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f032be00640 0 client.25583 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 shutdown_connections 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 1 -- 172.21.15.123:0/4236280193 wait complete. 2024-03-24T00:39:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.875+0000 7f034fa06740 20 client.25583 trim_cache size 0 max 16384 2024-03-24T00:39:37.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.892+0000 7f034fa06740 1 Processor -- start 2024-03-24T00:39:37.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.892+0000 7f034fa06740 1 -- start start 2024-03-24T00:39:37.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 0x560a555ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555c9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481a8a80 con 0x560a555ac2b0 2024-03-24T00:39:37.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481ab040 con 0x560a555b1e10 2024-03-24T00:39:37.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f034fa06740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481a9080 con 0x560a555acca0 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0345e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 0x560a555ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0345e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 0x560a555ab0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53618/0 (socket says 172.21.15.123:53618) 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0345e00640 1 -- 172.21.15.123:0/4023067049 learned_addr learned my addr 172.21.15.123:0/4023067049 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f033d400640 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 -- 172.21.15.123:0/4023067049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 msgr2=0x560a555c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 -- 172.21.15.123:0/4023067049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 msgr2=0x560a555ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 0x560a555ab0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.893+0000 7f0346800640 1 -- 172.21.15.123:0/4023067049 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a555a5fa0 con 0x560a555b1e10 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.894+0000 7f0346800640 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555c9550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0340037040 tx=0x7f034002e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb843659dc0a918d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.894+0000 7f0336a00640 1 -- 172.21.15.123:0/4023067049 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034005a070 con 0x560a555b1e10 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.894+0000 7f0336a00640 1 -- 172.21.15.123:0/4023067049 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0340003c30 con 0x560a555b1e10 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.894+0000 7f0336a00640 1 -- 172.21.15.123:0/4023067049 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340043040 con 0x560a555b1e10 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- 172.21.15.123:0/4023067049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 msgr2=0x560a555c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555c9550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0340037040 tx=0x7f034002e000 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- 172.21.15.123:0/4023067049 shutdown_connections 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555c9550 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555acca0 0x560a555c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 --2- 172.21.15.123:0/4023067049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555ac2b0 0x560a555ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- 172.21.15.123:0/4023067049 >> 172.21.15.123:0/4023067049 conn(0x560a555a7820 msgr2=0x560a555a6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- 172.21.15.123:0/4023067049 shutdown_connections 2024-03-24T00:39:37.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- 172.21.15.123:0/4023067049 wait complete. 2024-03-24T00:39:37.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.895+0000 7f034fa06740 1 -- start start 2024-03-24T00:39:37.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 Processor -- start 2024-03-24T00:39:37.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555ac2b0 0x560a555bcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555acca0 0x560a555bb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481a96c0 con 0x560a555acca0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481aa9e0 con 0x560a555b1e10 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481a9d20 con 0x560a555ac2b0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f034fa06740 10 client.0 _ll_register_callbacks cb 0x7f0338009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f0346800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f033d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555acca0 0x560a555bb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f0345e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555ac2b0 0x560a555bcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f0346800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54792/0 (socket says 172.21.15.123:54792) 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.896+0000 7f0346800640 1 -- 172.21.15.123:0/3135772772 learned_addr learned my addr 172.21.15.123:0/3135772772 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 -- 172.21.15.123:0/3135772772 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555ac2b0 msgr2=0x560a555bcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555ac2b0 0x560a555bcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 -- 172.21.15.123:0/3135772772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555acca0 msgr2=0x560a555bb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555acca0 0x560a555bb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 -- 172.21.15.123:0/3135772772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a555bbcf0 con 0x560a555b1e10 2024-03-24T00:39:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f0346800640 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0340033000 tx=0x7f0340009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8799cf36e7ab2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f032be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034005a070 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f03400045c0 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0340003c30 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.897+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0340043040 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 10 client.25589 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560a5561e860 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0340003c30 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f034005f950 con 0x560a555b1e10 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 10 client.25589 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 10 client.25589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 20 client.25589 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a5561eca0 0x560a55621090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 4 client.25589 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:39:37.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560a55519740 con 0x560a5561eca0 2024-03-24T00:39:37.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.898+0000 7f0345e00640 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a5561eca0 0x560a55621090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.899+0000 7f0345e00640 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a5561eca0 0x560a55621090 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a01f02ee2779543b server_cookie=ddb7375a584cc393 in_seq=0 out_seq=0 2024-03-24T00:39:37.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.899+0000 7f032be00640 10 client.25589 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.900+0000 7f032be00640 1 -- 172.21.15.123:0/3135772772 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560a55519740 con 0x560a5561eca0 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.900+0000 7f032be00640 10 client.25589 handle_command_reply: tid=0 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.900+0000 7f034fa06740 1 client.25589 shutdown 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.900+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 msgr2=0x560a555bb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.900+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0340033000 tx=0x7f0340009800 comp rx=0 tx=0).stop 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 shutdown_connections 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a555b1e10 0x560a555bb7b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560a555acca0 0x560a555bb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560a5561eca0 0x560a55621090 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 --2- 172.21.15.123:0/3135772772 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560a555ac2b0 0x560a555bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 >> 172.21.15.123:0/3135772772 conn(0x560a555a7820 msgr2=0x560a555c9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f032be00640 0 client.25589 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 shutdown_connections 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 1 -- 172.21.15.123:0/3135772772 wait complete. 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.901+0000 7f034fa06740 20 client.25589 trim_cache size 0 max 16384 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "aa202d9a-c8b3-4fe0-92f6-2382f208130a", 2024-03-24T00:39:37.904 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:39:37.905 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:37.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 msgr2=0x7f032c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 0x7f032c0a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0340032420 tx=0x7f034002c100 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 msgr2=0x7f0348073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f033000e9f0 tx=0x7f033000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 shutdown_connections 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f032c0a2800 0x7f032c0a4cc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0348103150 0x7f0348073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0348102790 0x7f0348073a90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 --2- 172.21.15.123:0/2608245411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03481015a0 0x7f0348075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 >> 172.21.15.123:0/2608245411 conn(0x7f03480fcb10 msgr2=0x7f0348105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 shutdown_connections 2024-03-24T00:39:37.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:37.902+0000 7f0347200640 1 -- 172.21.15.123:0/2608245411 wait complete. 2024-03-24T00:39:37.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:39:37.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:39:37.925 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether the C compiler works... yes 2024-03-24T00:39:37.925 INFO:tasks.workunit.client.0.smithi123.stdout:checking for C compiler default output file name... a.out 2024-03-24T00:39:38.009 INFO:tasks.workunit.client.0.smithi123.stdout:checking for suffix of executables... 2024-03-24T00:39:38.101 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are cross compiling... no 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.120+0000 7fa2e5000640 1 -- 172.21.15.123:0/4176887209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0105470 msgr2=0x7fa2e010bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 --2- 172.21.15.123:0/4176887209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0105470 0x7fa2e010bd00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009900 tx=0x7fa2d802bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 -- 172.21.15.123:0/4176887209 shutdown_connections 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 --2- 172.21.15.123:0/4176887209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0105470 0x7fa2e010bd00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 --2- 172.21.15.123:0/4176887209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e0104ab0 0x7fa2e0104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 --2- 172.21.15.123:0/4176887209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e01038c0 0x7fa2e0103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 -- 172.21.15.123:0/4176887209 >> 172.21.15.123:0/4176887209 conn(0x7fa2e00fee50 msgr2=0x7fa2e0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 -- 172.21.15.123:0/4176887209 shutdown_connections 2024-03-24T00:39:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.121+0000 7fa2e5000640 1 -- 172.21.15.123:0/4176887209 wait complete. 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 Processor -- start 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 -- start start 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e01038c0 0x7fa2e0199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 0x7fa2e01a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e010f570 con 0x7fa2e01038c0 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e010f3f0 con 0x7fa2e0104ab0 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2e5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e010f6f0 con 0x7fa2e0105470 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e01038c0 0x7fa2e0199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54806/0 (socket says 172.21.15.123:54806) 2024-03-24T00:39:38.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.122+0000 7fa2d5400640 1 -- 172.21.15.123:0/609703921 learned_addr learned my addr 172.21.15.123:0/609703921 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2de800640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 0x7fa2e01a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 -- 172.21.15.123:0/609703921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 msgr2=0x7fa2e01a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 0x7fa2e01a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 -- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e01038c0 msgr2=0x7fa2e0199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e01038c0 0x7fa2e0199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 -- 172.21.15.123:0/609703921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e01a1740 con 0x7fa2e0104ab0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2de800640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 0x7fa2e01a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d5400640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800e9f0 tx=0x7fa2c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d7400640 1 -- 172.21.15.123:0/609703921 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800cd80 con 0x7fa2e0104ab0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d7400640 1 -- 172.21.15.123:0/609703921 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2c8010910 con 0x7fa2e0104ab0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2d7400640 1 -- 172.21.15.123:0/609703921 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8018990 con 0x7fa2e0104ab0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.123+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e01a1a30 con 0x7fa2e0104ab0 2024-03-24T00:39:38.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.124+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e01a1f30 con 0x7fa2e0104ab0 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 Processor -- start 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2d7400640 1 -- 172.21.15.123:0/609703921 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa2c8010ac0 con 0x7fa2e0104ab0 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 -- start start 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b76370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b78e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916ad6250 con 0x55e916b70e80 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916a7ea30 con 0x55e916b71870 2024-03-24T00:39:38.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916ad22f0 con 0x55e916b769e0 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.127+0000 7fa2de800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53644/0 (socket says 172.21.15.123:53644) 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 -- 172.21.15.123:0/3330435339 learned_addr learned my addr 172.21.15.123:0/3330435339 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2d5400640 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b78e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2dde00640 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b76370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 -- 172.21.15.123:0/3330435339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 msgr2=0x55e916b78e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b78e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 -- 172.21.15.123:0/3330435339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 msgr2=0x55e916b76370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b76370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 -- 172.21.15.123:0/3330435339 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e916a69690 con 0x55e916b70e80 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.128+0000 7fa2de800640 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2d802c240 tx=0x7fa2d8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e45caf5d741067 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2d7400640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 0x7fa2c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2cf400640 1 -- 172.21.15.123:0/3330435339 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8044020 con 0x55e916b70e80 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2cf400640 1 -- 172.21.15.123:0/3330435339 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d802cb80 con 0x55e916b70e80 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2cf400640 1 -- 172.21.15.123:0/3330435339 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8004920 con 0x55e916b70e80 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2d7400640 1 -- 172.21.15.123:0/609703921 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2c80cee10 con 0x7fa2e0104ab0 2024-03-24T00:39:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.129+0000 7fa2dde00640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 0x7fa2c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 -- 172.21.15.123:0/3330435339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 msgr2=0x55e916b712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2dde00640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 0x7fa2c40a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55e916b71c70 tx=0x7fa2d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2d802c240 tx=0x7fa2d8009560 comp rx=0 tx=0).stop 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 -- 172.21.15.123:0/3330435339 shutdown_connections 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b78e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b76370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/3330435339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b712a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 -- 172.21.15.123:0/3330435339 >> 172.21.15.123:0/3330435339 conn(0x55e916b39200 msgr2=0x55e916b6c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 -- 172.21.15.123:0/3330435339 shutdown_connections 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.130+0000 7fa2e7c13740 1 -- 172.21.15.123:0/3330435339 wait complete. 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 -- start start 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 Processor -- start 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b7b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b82350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916b79930 con 0x55e916b70e80 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916a91e80 con 0x55e916b71870 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e916b79ab0 con 0x55e916b769e0 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.131+0000 7fa2e7c13740 10 client.0 _ll_register_callbacks cb 0x55e916aa4130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:38.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53648/0 (socket says 172.21.15.123:53648) 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b82350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2dde00640 1 -- 172.21.15.123:0/183786936 learned_addr learned my addr 172.21.15.123:0/183786936 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2d5400640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b7b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 -- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 msgr2=0x55e916b7b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b7b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 -- 172.21.15.123:0/183786936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 msgr2=0x55e916b7add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 -- 172.21.15.123:0/183786936 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9168b47d0 con 0x55e916b769e0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2dde00640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2de800640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b82350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8036040 tx=0x7fa2d8040bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d3215fef1e198a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.132+0000 7fa2c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d804d030 con 0x55e916b769e0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2d802c980 con 0x55e916b769e0 2024-03-24T00:39:38.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2d802cb80 con 0x55e916b769e0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2d8044040 con 0x55e916b769e0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2e7c13740 10 client.25588 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e9167ea3f0 con 0x55e916b769e0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d802cb80 con 0x55e916b769e0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa2d8034440 con 0x55e916b769e0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2e7c13740 10 client.25588 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.133+0000 7fa2e7c13740 10 client.25588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.134+0000 7fa2e7c13740 20 client.25588 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.134+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916b86840 0x55e916b88c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.134+0000 7fa2e7c13740 4 client.25588 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.134+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e916ae3790 con 0x55e916b86840 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.134+0000 7fa2dde00640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916b86840 0x55e916b88c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2c3e00640 10 client.25588 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2dde00640 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916b86840 0x55e916b88c30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=276b8024e7a6a56e server_cookie=964e2acef282b28e in_seq=0 out_seq=0 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2c3e00640 1 -- 172.21.15.123:0/183786936 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e916ae3790 con 0x55e916b86840 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2c3e00640 10 client.25588 handle_command_reply: tid=0 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2e7c13740 1 client.25588 shutdown 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 msgr2=0x55e916b82350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.135+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b82350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8036040 tx=0x7fa2d8040bf0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 shutdown_connections 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916b86840 0x55e916b88c30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b769e0 0x55e916b82350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b71870 0x55e916b7b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/183786936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2c3e00640 0 client.25588 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 >> 172.21.15.123:0/183786936 conn(0x55e916b39200 msgr2=0x55e916b6ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 shutdown_connections 2024-03-24T00:39:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 1 -- 172.21.15.123:0/183786936 wait complete. 2024-03-24T00:39:38.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.136+0000 7fa2e7c13740 20 client.25588 trim_cache size 0 max 16384 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.153+0000 7fa2e7c13740 1 Processor -- start 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 -- start start 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 0x55e916b7a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b71870 0x55e916b82310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01a8d00 con 0x55e916b71870 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01ab2c0 con 0x55e916b769e0 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01a9300 con 0x55e916b70e80 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2de800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 0x55e916b7a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2de800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54830/0 (socket says 172.21.15.123:54830) 2024-03-24T00:39:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2de800640 1 -- 172.21.15.123:0/887004041 learned_addr learned my addr 172.21.15.123:0/887004041 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2d5400640 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b71870 0x55e916b82310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2de800640 1 -- 172.21.15.123:0/887004041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 msgr2=0x55e916b7a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.154+0000 7fa2de800640 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 0x55e916b7a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2de800640 1 -- 172.21.15.123:0/887004041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b71870 msgr2=0x55e916b82310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2de800640 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b71870 0x55e916b82310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2de800640 1 -- 172.21.15.123:0/887004041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e916b763b0 con 0x55e916b769e0 2024-03-24T00:39:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2dde00640 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 0x55e916b7a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2de800640 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa2d802c7b0 tx=0x7fa2d8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad4a6ba12ccd0b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2cea00640 1 -- 172.21.15.123:0/887004041 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d804d030 con 0x55e916b769e0 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2cea00640 1 -- 172.21.15.123:0/887004041 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8034440 con 0x55e916b769e0 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.155+0000 7fa2cea00640 1 -- 172.21.15.123:0/887004041 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8005030 con 0x55e916b769e0 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 -- 172.21.15.123:0/887004041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 msgr2=0x55e916b729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa2d802c7b0 tx=0x7fa2d8000bc0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 -- 172.21.15.123:0/887004041 shutdown_connections 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b729d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b71870 0x55e916b82310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/887004041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b70e80 0x55e916b7a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.156+0000 7fa2e7c13740 1 -- 172.21.15.123:0/887004041 >> 172.21.15.123:0/887004041 conn(0x55e916b39200 msgr2=0x55e916b722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.157+0000 7fa2e7c13740 1 -- 172.21.15.123:0/887004041 shutdown_connections 2024-03-24T00:39:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.157+0000 7fa2e7c13740 1 -- 172.21.15.123:0/887004041 wait complete. 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.157+0000 7fa2e7c13740 1 -- start start 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.157+0000 7fa2e7c13740 1 Processor -- start 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b71870 0x55e916b80200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b811a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01a9940 con 0x55e916b70e80 2024-03-24T00:39:38.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01aac60 con 0x55e916b769e0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01a9fa0 con 0x55e916b71870 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2e7c13740 10 client.0 _ll_register_callbacks cb 0x7fa2d0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53658/0 (socket says 172.21.15.123:53658) 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2dde00640 1 -- 172.21.15.123:0/4077779840 learned_addr learned my addr 172.21.15.123:0/4077779840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2de800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.158+0000 7fa2d5400640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b71870 0x55e916b80200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 -- 172.21.15.123:0/4077779840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b71870 msgr2=0x55e916b80200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b71870 0x55e916b80200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 -- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 msgr2=0x55e916b811a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b811a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 -- 172.21.15.123:0/4077779840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9167aa0f0 con 0x55e916b70e80 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2de800640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b811a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2dde00640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0005fd0 tx=0x7fa2d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44cc310a6053e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0004b20 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2d0004cc0 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.159+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2d8034440 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2d00042d0 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 10 client.15501 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e916be1fe0 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8034440 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa2d002acb0 con 0x55e916b70e80 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 10 client.15501 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 10 client.15501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 20 client.15501 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916be2910 0x55e916be4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 4 client.15501 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.160+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e916b7abc0 con 0x55e916be2910 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2d5400640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916be2910 0x55e916be4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2d5400640 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916be2910 0x55e916be4dd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97b6117605962ac server_cookie=60aca08b480f3496 in_seq=0 out_seq=0 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2c3e00640 10 client.15501 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2c3e00640 1 -- 172.21.15.123:0/4077779840 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e916b7abc0 con 0x55e916be2910 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2c3e00640 10 client.15501 handle_command_reply: tid=0 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 client.15501 shutdown 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 msgr2=0x55e916b7fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0005fd0 tx=0x7fa2d0017000 comp rx=0 tx=0).stop 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 shutdown_connections 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e916b769e0 0x55e916b811a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e916be2910 0x55e916be4dd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e916b71870 0x55e916b80200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 --2- 172.21.15.123:0/4077779840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e916b70e80 0x55e916b7fcc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 >> 172.21.15.123:0/4077779840 conn(0x55e916b39200 msgr2=0x55e916b6b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.161+0000 7fa2c3e00640 0 client.15501 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 shutdown_connections 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e7c13740 1 -- 172.21.15.123:0/4077779840 wait complete. 2024-03-24T00:39:38.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e7c13740 20 client.15501 trim_cache size 0 max 16384 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "aa202d9a-c8b3-4fe0-92f6-2382f208130a": { 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "aa202d9a-c8b3-4fe0-92f6-2382f208130a", 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:39:38.166 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 msgr2=0x7fa2c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 0x7fa2c40a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55e916b71c70 tx=0x7fa2d0009500 comp rx=0 tx=0).stop 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 msgr2=0x7fa2e019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.162+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800e9f0 tx=0x7fa2c800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 shutdown_connections 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0105470 0x7fa2e01a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2e0104ab0 0x7fa2e019a060 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2c40a2880 0x7fa2c40a4d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 --2- 172.21.15.123:0/609703921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e01038c0 0x7fa2e0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 >> 172.21.15.123:0/609703921 conn(0x7fa2e00fee50 msgr2=0x7fa2e0107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 shutdown_connections 2024-03-24T00:39:38.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:39:38.163+0000 7fa2e5000640 1 -- 172.21.15.123:0/609703921 wait complete. 2024-03-24T00:39:38.180 INFO:tasks.cephfs.filesystem:scrub status for tag:aa202d9a-c8b3-4fe0-92f6-2382f208130a - {'path': '/', 'tag': 'aa202d9a-c8b3-4fe0-92f6-2382f208130a', 'options': 'recursive,force'} 2024-03-24T00:39:38.183 INFO:tasks.workunit.client.0.smithi123.stdout:checking for suffix of object files... o 2024-03-24T00:39:38.249 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T00:39:38.332 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether gcc accepts -g... yes 2024-03-24T00:39:38.437 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-24T00:39:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:38 smithi204 ceph-mon[36345]: pgmap v437: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 363 KiB/s wr, 149 op/s 2024-03-24T00:39:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:38 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:39:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:38 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:38 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:39:38.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:38 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:38 smithi123 ceph-mon[31487]: pgmap v437: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 363 KiB/s wr, 149 op/s 2024-03-24T00:39:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:38 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:39:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:38 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:38 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:39:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:38 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.682 INFO:tasks.workunit.client.0.smithi123.stdout:checking dependency style of gcc... gcc3 2024-03-24T00:39:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:38 smithi177 ceph-mon[36034]: pgmap v437: 97 pgs: 97 active+clean; 4.6 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 363 KiB/s wr, 149 op/s 2024-03-24T00:39:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:38 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:39:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:38 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:38 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:39:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:38 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:39:38.820 INFO:tasks.workunit.client.0.smithi123.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T00:39:38.962 INFO:tasks.workunit.client.0.smithi123.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-24T00:39:38.968 INFO:tasks.workunit.client.0.smithi123.stdout:checking for egrep... /usr/bin/grep -E 2024-03-24T00:39:39.267 INFO:tasks.workunit.client.0.smithi123.stdout:checking for ANSI C header files... yes 2024-03-24T00:39:39.360 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/types.h... yes 2024-03-24T00:39:39.557 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/stat.h... yes 2024-03-24T00:39:39.653 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdlib.h... yes 2024-03-24T00:39:39.756 INFO:tasks.workunit.client.0.smithi123.stdout:checking for string.h... yes 2024-03-24T00:39:39.866 INFO:tasks.workunit.client.0.smithi123.stdout:checking for memory.h... yes 2024-03-24T00:39:39.971 INFO:tasks.workunit.client.0.smithi123.stdout:checking for strings.h... yes 2024-03-24T00:39:40.079 INFO:tasks.workunit.client.0.smithi123.stdout:checking for inttypes.h... yes 2024-03-24T00:39:40.183 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdint.h... yes 2024-03-24T00:39:40.294 INFO:tasks.workunit.client.0.smithi123.stdout:checking for unistd.h... yes 2024-03-24T00:39:40.397 INFO:tasks.workunit.client.0.smithi123.stdout:checking minix/config.h usability... no 2024-03-24T00:39:40.500 INFO:tasks.workunit.client.0.smithi123.stdout:checking minix/config.h presence... no 2024-03-24T00:39:40.500 INFO:tasks.workunit.client.0.smithi123.stdout:checking for minix/config.h... no 2024-03-24T00:39:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:40 smithi204 ceph-mon[36345]: pgmap v438: 97 pgs: 97 active+clean; 4.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 363 KiB/s wr, 160 op/s 2024-03-24T00:39:40.584 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-24T00:39:40.586 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc... (cached) gcc 2024-03-24T00:39:40.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:40 smithi123 ceph-mon[31487]: pgmap v438: 97 pgs: 97 active+clean; 4.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 363 KiB/s wr, 160 op/s 2024-03-24T00:39:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:40 smithi177 ceph-mon[36034]: pgmap v438: 97 pgs: 97 active+clean; 4.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 363 KiB/s wr, 160 op/s 2024-03-24T00:39:40.713 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-24T00:39:40.714 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-24T00:39:40.714 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-24T00:39:40.715 INFO:tasks.workunit.client.0.smithi123.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-24T00:39:40.843 INFO:tasks.workunit.client.0.smithi123.stdout:checking for main in -lm... yes 2024-03-24T00:39:40.966 INFO:tasks.workunit.client.0.smithi123.stdout:checking for main in -lpthread... yes 2024-03-24T00:39:40.970 INFO:tasks.workunit.client.0.smithi123.stdout:checking for ANSI C header files... (cached) yes 2024-03-24T00:39:41.067 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-24T00:39:41.189 INFO:tasks.workunit.client.0.smithi123.stdout:checking pthread.h usability... yes 2024-03-24T00:39:41.249 INFO:tasks.workunit.client.0.smithi123.stdout:checking pthread.h presence... yes 2024-03-24T00:39:41.249 INFO:tasks.workunit.client.0.smithi123.stdout:checking for pthread.h... yes 2024-03-24T00:39:41.369 INFO:tasks.workunit.client.0.smithi123.stdout:checking fcntl.h usability... yes 2024-03-24T00:39:41.426 INFO:tasks.workunit.client.0.smithi123.stdout:checking fcntl.h presence... yes 2024-03-24T00:39:41.426 INFO:tasks.workunit.client.0.smithi123.stdout:checking for fcntl.h... yes 2024-03-24T00:39:41.576 INFO:tasks.workunit.client.0.smithi123.stdout:checking limits.h usability... yes 2024-03-24T00:39:41.628 INFO:tasks.workunit.client.0.smithi123.stdout:checking limits.h presence... yes 2024-03-24T00:39:41.628 INFO:tasks.workunit.client.0.smithi123.stdout:checking for limits.h... yes 2024-03-24T00:39:41.650 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stdint.h... (cached) yes 2024-03-24T00:39:41.762 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/time.h usability... yes 2024-03-24T00:39:41.816 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/time.h presence... yes 2024-03-24T00:39:41.816 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/time.h... yes 2024-03-24T00:39:41.840 INFO:tasks.workunit.client.0.smithi123.stdout:checking for unistd.h... (cached) yes 2024-03-24T00:39:41.941 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/vfs.h usability... yes 2024-03-24T00:39:41.998 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/vfs.h presence... yes 2024-03-24T00:39:41.998 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/vfs.h... yes 2024-03-24T00:39:42.118 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/limits.h usability... no 2024-03-24T00:39:42.196 INFO:tasks.workunit.client.0.smithi123.stdout:checking sys/limits.h presence... no 2024-03-24T00:39:42.196 INFO:tasks.workunit.client.0.smithi123.stdout:checking for sys/limits.h... no 2024-03-24T00:39:42.276 INFO:tasks.workunit.client.0.smithi123.stdout:checking for an ANSI C-conforming const... yes 2024-03-24T00:39:42.370 INFO:tasks.workunit.client.0.smithi123.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-24T00:39:42.503 INFO:tasks.workunit.client.0.smithi123.stdout:checking for system... yes 2024-03-24T00:39:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:42 smithi204 ceph-mon[36345]: pgmap v439: 97 pgs: 97 active+clean; 6.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 548 KiB/s wr, 206 op/s 2024-03-24T00:39:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:42 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:39:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:42 smithi123 ceph-mon[31487]: pgmap v439: 97 pgs: 97 active+clean; 6.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 548 KiB/s wr, 206 op/s 2024-03-24T00:39:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:42 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:39:42.641 INFO:tasks.workunit.client.0.smithi123.stdout:checking for gettimeofday... yes 2024-03-24T00:39:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:42 smithi177 ceph-mon[36034]: pgmap v439: 97 pgs: 97 active+clean; 6.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 548 KiB/s wr, 206 op/s 2024-03-24T00:39:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:42 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:39:42.779 INFO:tasks.workunit.client.0.smithi123.stdout:checking for mkdir... yes 2024-03-24T00:39:42.917 INFO:tasks.workunit.client.0.smithi123.stdout:checking for strerror... yes 2024-03-24T00:39:43.067 INFO:tasks.workunit.client.0.smithi123.stdout:checking for open64... yes 2024-03-24T00:39:43.206 INFO:tasks.workunit.client.0.smithi123.stdout:checking for stat64... yes 2024-03-24T00:39:43.347 INFO:tasks.workunit.client.0.smithi123.stdout:checking for fseeko64... yes 2024-03-24T00:39:43.490 INFO:tasks.workunit.client.0.smithi123.stdout:checking for lrand48_r... yes 2024-03-24T00:39:43.629 INFO:tasks.workunit.client.0.smithi123.stdout:checking for srand48_r... yes 2024-03-24T00:39:43.679 INFO:tasks.workunit.client.0.smithi123.stdout:configure: creating ./config.status 2024-03-24T00:39:43.847 INFO:tasks.workunit.client.0.smithi123.stdout: /bin/sh ./config.status 2024-03-24T00:39:44.003 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: creating Makefile 2024-03-24T00:39:44.057 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: creating config.h 2024-03-24T00:39:44.113 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: executing depfiles commands 2024-03-24T00:39:44.164 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: executing default commands 2024-03-24T00:39:44.218 INFO:tasks.workunit.client.0.smithi123.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-24T00:39:44.480 INFO:tasks.workunit.client.0.smithi123.stdout:rm -f stamp-h1 2024-03-24T00:39:44.488 INFO:tasks.workunit.client.0.smithi123.stdout:touch config.h.in 2024-03-24T00:39:44.494 INFO:tasks.workunit.client.0.smithi123.stdout:cd . && /bin/sh ./config.status config.h 2024-03-24T00:39:44.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:44 smithi204 ceph-mon[36345]: pgmap v440: 97 pgs: 97 active+clean; 6.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 369 KiB/s wr, 185 op/s 2024-03-24T00:39:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:44 smithi123 ceph-mon[31487]: pgmap v440: 97 pgs: 97 active+clean; 6.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 369 KiB/s wr, 185 op/s 2024-03-24T00:39:44.627 INFO:tasks.workunit.client.0.smithi123.stdout:config.status: creating config.h 2024-03-24T00:39:44.694 INFO:tasks.workunit.client.0.smithi123.stdout:make all-am 2024-03-24T00:39:44.705 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-24T00:39:44.705 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-24T00:39:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:44 smithi177 ceph-mon[36034]: pgmap v440: 97 pgs: 97 active+clean; 6.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 369 KiB/s wr, 185 op/s 2024-03-24T00:39:44.993 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-24T00:39:45.001 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-24T00:39:45.175 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-24T00:39:45.185 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-24T00:39:45.356 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-24T00:39:45.366 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-24T00:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 -- 172.21.15.123:0/2501957595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0104900 msgr2=0x7f4bb0106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 --2- 172.21.15.123:0/2501957595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0104900 0x7f4bb0106cf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009900 tx=0x7f4b9c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 -- 172.21.15.123:0/2501957595 shutdown_connections 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 --2- 172.21.15.123:0/2501957595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0107230 0x7f4bb0109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 --2- 172.21.15.123:0/2501957595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0104900 0x7f4bb0106cf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 --2- 172.21.15.123:0/2501957595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0101fd0 0x7f4bb01043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 -- 172.21.15.123:0/2501957595 >> 172.21.15.123:0/2501957595 conn(0x7f4bb00fb980 msgr2=0x7f4bb00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.432+0000 7f4bb5200640 1 -- 172.21.15.123:0/2501957595 shutdown_connections 2024-03-24T00:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.433+0000 7f4bb5200640 1 -- 172.21.15.123:0/2501957595 wait complete. 2024-03-24T00:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.433+0000 7f4bb5200640 1 Processor -- start 2024-03-24T00:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 -- start start 2024-03-24T00:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0104900 0x7f4bb00fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0107230 0x7f4bb01004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb010d1d0 con 0x7f4bb0104900 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb010d050 con 0x7f4bb0101fd0 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bb5200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb010d350 con 0x7f4bb0107230 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4baf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4baf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54856/0 (socket says 172.21.15.123:54856) 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4baf400640 1 -- 172.21.15.123:0/3052582485 learned_addr learned my addr 172.21.15.123:0/3052582485 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4baea00640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0104900 0x7f4bb00fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.434+0000 7f4bafe00640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0107230 0x7f4bb01004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 -- 172.21.15.123:0/3052582485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0107230 msgr2=0x7f4bb01004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0107230 0x7f4bb01004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 -- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0104900 msgr2=0x7f4bb00fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0104900 0x7f4bb00fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 -- 172.21.15.123:0/3052582485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb0100d50 con 0x7f4bb0101fd0 2024-03-24T00:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4baf400640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4ba7e00640 1 -- 172.21.15.123:0/3052582485 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000cd80 con 0x7f4bb0101fd0 2024-03-24T00:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4ba7e00640 1 -- 172.21.15.123:0/3052582485 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0005500 con 0x7f4bb0101fd0 2024-03-24T00:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4ba7e00640 1 -- 172.21.15.123:0/3052582485 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0010690 con 0x7f4bb0101fd0 2024-03-24T00:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb01a1970 con 0x7f4bb0101fd0 2024-03-24T00:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.435+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb01a1e90 con 0x7f4bb0101fd0 2024-03-24T00:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.438+0000 7f4ba7e00640 1 -- 172.21.15.123:0/3052582485 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4ba00046c0 con 0x7f4bb0101fd0 2024-03-24T00:39:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.439+0000 7f4ba7e00640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 0x7f4b940a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.440+0000 7f4ba7e00640 1 -- 172.21.15.123:0/3052582485 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4ba00cee10 con 0x7f4bb0101fd0 2024-03-24T00:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.440+0000 7f4baea00640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 0x7f4b940a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.440+0000 7f4bb9638740 1 Processor -- start 2024-03-24T00:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.440+0000 7f4bb9638740 1 -- start start 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c677810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 0x55e92c67c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c63a3b0 con 0x55e92c6773f0 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c5dc830 con 0x55e92c67cf50 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bb9638740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c5855d0 con 0x55e92c677de0 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4baea00640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 0x7f4b940a4d30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02c170 tx=0x7f4b9c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4baf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c677810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54872/0 (socket says 172.21.15.123:54872) 2024-03-24T00:39:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 -- 172.21.15.123:0/1275943146 learned_addr learned my addr 172.21.15.123:0/1275943146 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4baea00640 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 0x55e92c67c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 -- 172.21.15.123:0/1275943146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 msgr2=0x55e92c67c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 0x55e92c67c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 -- 172.21.15.123:0/1275943146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 msgr2=0x55e92c677810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c677810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4bafe00640 1 -- 172.21.15.123:0/1275943146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e92c59d2c0 con 0x55e92c67cf50 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.441+0000 7f4baea00640 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 0x55e92c67c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.442+0000 7f4bafe00640 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80039f0 tx=0x7f4ba8003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7f0509064d7cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.442+0000 7f4ba4c00640 1 -- 172.21.15.123:0/1275943146 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba801a470 con 0x55e92c67cf50 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.442+0000 7f4ba4c00640 1 -- 172.21.15.123:0/1275943146 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba8015070 con 0x55e92c67cf50 2024-03-24T00:39:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.442+0000 7f4ba4c00640 1 -- 172.21.15.123:0/1275943146 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8010900 con 0x55e92c67cf50 2024-03-24T00:39:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 -- 172.21.15.123:0/1275943146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 msgr2=0x55e92c67f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80039f0 tx=0x7f4ba8003eb0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 -- 172.21.15.123:0/1275943146 shutdown_connections 2024-03-24T00:39:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c67f3e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c677de0 0x55e92c67c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1275943146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c677810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 -- 172.21.15.123:0/1275943146 >> 172.21.15.123:0/1275943146 conn(0x55e92c672960 msgr2=0x55e92c672d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 -- 172.21.15.123:0/1275943146 shutdown_connections 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.443+0000 7f4bb9638740 1 -- 172.21.15.123:0/1275943146 wait complete. 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.444+0000 7f4bb9638740 1 -- start start 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.444+0000 7f4bb9638740 1 Processor -- start 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c681ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c688b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c598a20 con 0x55e92c677de0 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c5d88d0 con 0x55e92c6773f0 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92c67ff00 con 0x55e92c67cf50 2024-03-24T00:39:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bb9638740 10 client.0 _ll_register_callbacks cb 0x55e92c5d51c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54878/0 (socket says 172.21.15.123:54878) 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 -- 172.21.15.123:0/911958313 learned_addr learned my addr 172.21.15.123:0/911958313 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baea00640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c681ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4bafe00640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c688b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 -- 172.21.15.123:0/911958313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 msgr2=0x55e92c688b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c688b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 -- 172.21.15.123:0/911958313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 msgr2=0x55e92c681ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c681ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.445+0000 7f4baf400640 1 -- 172.21.15.123:0/911958313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e92c2f1210 con 0x55e92c6773f0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4baf400640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9c0 tx=0x7f4ba009b100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96801f2511272f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4b99600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00b3020 con 0x55e92c6773f0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4ba00a3ab0 con 0x55e92c6773f0 2024-03-24T00:39:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ba8015070 con 0x55e92c6773f0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba009f8f0 con 0x55e92c6773f0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4bb9638740 10 client.25619 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e92c3bd620 con 0x55e92c6773f0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.446+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba8015070 con 0x55e92c6773f0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4ba009abf0 con 0x55e92c6773f0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 10 client.25619 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 10 client.25619 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 20 client.25619 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c68d460 0x55e92c68f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 4 client.25619 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e92c5e9d70 con 0x55e92c68d460 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4baea00640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c68d460 0x55e92c68f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4baea00640 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c68d460 0x55e92c68f850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e208e57263ff62f1 server_cookie=8b05be87202a822b in_seq=0 out_seq=0 2024-03-24T00:39:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.447+0000 7f4b99600640 10 client.25619 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4b99600640 1 -- 172.21.15.123:0/911958313 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e92c5e9d70 con 0x55e92c68d460 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4b99600640 10 client.25619 handle_command_reply: tid=0 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 client.25619 shutdown 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 msgr2=0x55e92c6815a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9c0 tx=0x7f4ba009b100 comp rx=0 tx=0).stop 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 shutdown_connections 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c68d460 0x55e92c68f850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c688b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4b99600640 0 client.25619 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c681ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 --2- 172.21.15.123:0/911958313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c6773f0 0x55e92c6815a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.448+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 >> 172.21.15.123:0/911958313 conn(0x55e92c672960 msgr2=0x55e92c671b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.449+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 shutdown_connections 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.449+0000 7f4bb9638740 1 -- 172.21.15.123:0/911958313 wait complete. 2024-03-24T00:39:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.449+0000 7f4bb9638740 20 client.25619 trim_cache size 0 max 16384 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 Processor -- start 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 -- start start 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c6787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c672d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01a8b70 con 0x55e92c6773f0 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01ab130 con 0x55e92c677de0 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4bb9638740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01a9170 con 0x55e92c67cf50 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4baea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4baea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54888/0 (socket says 172.21.15.123:54888) 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4baea00640 1 -- 172.21.15.123:0/3459908443 learned_addr learned my addr 172.21.15.123:0/3459908443 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.464+0000 7f4baf400640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c6787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4bafe00640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c672d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 -- 172.21.15.123:0/3459908443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 msgr2=0x55e92c672d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c672d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 -- 172.21.15.123:0/3459908443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 msgr2=0x55e92c6787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c6787e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 -- 172.21.15.123:0/3459908443 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e92c6736d0 con 0x55e92c677de0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baf400640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c6787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4baea00640 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02c240 tx=0x7f4b9c03f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4096cb44559bd681 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4b9be00640 1 -- 172.21.15.123:0/3459908443 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c05a070 con 0x55e92c677de0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4b9be00640 1 -- 172.21.15.123:0/3459908443 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b9c046040 con 0x55e92c677de0 2024-03-24T00:39:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.465+0000 7f4b9be00640 1 -- 172.21.15.123:0/3459908443 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c004c50 con 0x55e92c677de0 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 -- 172.21.15.123:0/3459908443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 msgr2=0x55e92c688ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02c240 tx=0x7f4b9c03f000 comp rx=0 tx=0).stop 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 -- 172.21.15.123:0/3459908443 shutdown_connections 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c67cf50 0x55e92c672d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c677de0 0x55e92c688ae0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 --2- 172.21.15.123:0/3459908443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c6773f0 0x55e92c6787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 -- 172.21.15.123:0/3459908443 >> 172.21.15.123:0/3459908443 conn(0x55e92c672960 msgr2=0x55e92c67c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 -- 172.21.15.123:0/3459908443 shutdown_connections 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.466+0000 7f4bb9638740 1 -- 172.21.15.123:0/3459908443 wait complete. 2024-03-24T00:39:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 -- start start 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 Processor -- start 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c688910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c6869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01a97b0 con 0x55e92c677de0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01aaad0 con 0x55e92c67cf50 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01a9e10 con 0x55e92c6773f0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bb9638740 10 client.0 _ll_register_callbacks cb 0x7f4ba0097a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4baf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.467+0000 7f4bafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c6869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c688910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37618/0 (socket says 172.21.15.123:37618) 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 -- 172.21.15.123:0/1655853549 learned_addr learned my addr 172.21.15.123:0/1655853549 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 -- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 msgr2=0x55e92c6869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c6869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 -- 172.21.15.123:0/1655853549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 msgr2=0x55e92c688910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c688910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 -- 172.21.15.123:0/1655853549 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e92c680040 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4bafe00640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c6869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4baf400640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0098120 tx=0x7f4ba0099cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2906b6754b4f4a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4b99600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00ad020 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4ba00afb80 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.468+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b9c046040 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba00b3040 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 10 client.25612 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e92c6e9510 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b9c046040 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4ba009a660 con 0x55e92c6773f0 2024-03-24T00:39:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 10 client.25612 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 10 client.25612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 20 client.25612 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c6e9e70 0x55e92c6ec290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 4 client.25612 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.469+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e92c5e40c0 con 0x55e92c6e9e70 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.470+0000 7f4baea00640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c6e9e70 0x55e92c6ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.470+0000 7f4b99600640 10 client.25612 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.470+0000 7f4baea00640 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c6e9e70 0x55e92c6ec290 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283a08d8d34e04a4 server_cookie=c1f0de3ad15b83d5 in_seq=0 out_seq=0 2024-03-24T00:39:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4b99600640 1 -- 172.21.15.123:0/1655853549 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e92c5e40c0 con 0x55e92c6e9e70 2024-03-24T00:39:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4b99600640 10 client.25612 handle_command_reply: tid=0 2024-03-24T00:39:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 client.25612 shutdown 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 msgr2=0x55e92c6883d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0098120 tx=0x7f4ba0099cb0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 shutdown_connections 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e92c67cf50 0x55e92c6869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e92c677de0 0x55e92c688910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e92c6e9e70 0x55e92c6ec290 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 --2- 172.21.15.123:0/1655853549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e92c6773f0 0x55e92c6883d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4b99600640 0 client.25612 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 >> 172.21.15.123:0/1655853549 conn(0x55e92c672960 msgr2=0x55e92c672d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 shutdown_connections 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 1 -- 172.21.15.123:0/1655853549 wait complete. 2024-03-24T00:39:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.523+0000 7f4bb9638740 20 client.25612 trim_cache size 0 max 16384 2024-03-24T00:39:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:39:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:39:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:39:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 msgr2=0x7f4b940a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 0x7f4b940a4d30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02c170 tx=0x7f4b9c02df90 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 msgr2=0x7f4bb0101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 shutdown_connections 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bb0107230 0x7f4bb01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4b940a2870 0x7f4b940a4d30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb0104900 0x7f4bb00fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 --2- 172.21.15.123:0/3052582485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb0101fd0 0x7f4bb0101940 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 >> 172.21.15.123:0/3052582485 conn(0x7f4bb00fb980 msgr2=0x7f4bb0102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 shutdown_connections 2024-03-24T00:39:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:45.524+0000 7f4bb5200640 1 -- 172.21.15.123:0/3052582485 wait complete. 2024-03-24T00:39:45.552 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-24T00:39:45.560 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-24T00:39:45.750 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-24T00:39:45.760 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-24T00:39:45.873 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-24T00:39:45.883 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-24T00:39:45.978 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-24T00:39:45.989 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-24T00:39:46.095 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-24T00:39:46.106 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-24T00:39:46.402 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-24T00:39:46.412 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-24T00:39:46.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:46 smithi204 ceph-mon[36345]: pgmap v441: 97 pgs: 97 active+clean; 6.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 356 KiB/s wr, 164 op/s 2024-03-24T00:39:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:46 smithi123 ceph-mon[31487]: pgmap v441: 97 pgs: 97 active+clean; 6.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 356 KiB/s wr, 164 op/s 2024-03-24T00:39:46.646 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-24T00:39:46.658 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-24T00:39:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:46 smithi177 ceph-mon[36034]: pgmap v441: 97 pgs: 97 active+clean; 6.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 356 KiB/s wr, 164 op/s 2024-03-24T00:39:47.273 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-24T00:39:47.285 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-24T00:39:47.431 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-24T00:39:47.441 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-24T00:39:47.586 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-24T00:39:47.598 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-24T00:39:47.781 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-24T00:39:47.792 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr:In file included from parser.h:22, 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: from parser.c:25: 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c: In function ‘init_filesys’: 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.873 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.874 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.875 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.876 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-24T00:39:47.877 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.878 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.879 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.880 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: | ^ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T00:39:47.881 INFO:tasks.workunit.client.0.smithi123.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: 36 | container_of(ptr, type, member) 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-24T00:39:47.882 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c: In function ‘get_optstr’: 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:200:14: note: source object declared here 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr: 200 | char temp[BUFSIZE]; 2024-03-24T00:39:47.953 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~ 2024-03-24T00:39:47.965 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c: In function ‘size64_convert’: 2024-03-24T00:39:47.965 INFO:tasks.workunit.client.0.smithi123.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-24T00:39:47.965 INFO:tasks.workunit.client.0.smithi123.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-24T00:39:47.965 INFO:tasks.workunit.client.0.smithi123.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-24T00:39:48.329 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-24T00:39:48.342 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-24T00:39:48.487 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-24T00:39:48.500 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-24T00:39:48.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:48 smithi204 ceph-mon[36345]: pgmap v442: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 125 op/s 2024-03-24T00:39:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:48 smithi123 ceph-mon[31487]: pgmap v442: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 125 op/s 2024-03-24T00:39:48.681 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-24T00:39:48.693 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-24T00:39:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:48 smithi177 ceph-mon[36034]: pgmap v442: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 125 op/s 2024-03-24T00:39:48.774 INFO:tasks.workunit.client.0.smithi123.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-24T00:39:48.789 INFO:tasks.workunit.client.0.smithi123.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-24T00:39:48.899 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-24T00:39:48.900 INFO:tasks.workunit.client.0.smithi123.stderr:+ cd .. 2024-03-24T00:39:48.900 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir tmp 2024-03-24T00:39:48.906 INFO:tasks.workunit.client.0.smithi123.stderr:+ cd tmp 2024-03-24T00:39:48.906 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in $mydir/*.ffsb 2024-03-24T00:39:48.907 INFO:tasks.workunit.client.0.smithi123.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-24T00:39:48.909 INFO:tasks.workunit.client.0.smithi123.stdout:FFSB version 6.0-RC2 started 2024-03-24T00:39:48.909 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout:benchmark time = 300 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout:ThreadGroup 0 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout:================ 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout: num_threads = 32 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout: read_random = off 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout: read_size = 0 (0B) 2024-03-24T00:39:48.917 INFO:tasks.workunit.client.0.smithi123.stdout: read_blocksize = 0 (0B) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: read_skip = off 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: read_skipsize = 0 (0B) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: write_random = on 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: write_size = 5242880 (5MB) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: fsync_file = 0 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: write_blocksize = 4096 (4KB) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: wait time = 0 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: op weights 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: read = 0 (0.00%) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: readall = 0 (0.00%) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: write = 1 (100.00%) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: create = 0 (0.00%) 2024-03-24T00:39:48.918 INFO:tasks.workunit.client.0.smithi123.stdout: append = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: delete = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: metaop = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: createdir = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: stat = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: writeall = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: writeall_fsync = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: open_close = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: write_fsync = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: create_fsync = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: append_fsync = 0 (0.00%) 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout:FileSystem . 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout:========== 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: num_dirs = 0 2024-03-24T00:39:48.919 INFO:tasks.workunit.client.0.smithi123.stdout: starting files = 128 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: min file size = 104857600 (100MB) 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: max file size = 104857600 (100MB) 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: directio = off 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: alignedio = on 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: bufferedio = off 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.920 INFO:tasks.workunit.client.0.smithi123.stdout: aging is off 2024-03-24T00:39:48.921 INFO:tasks.workunit.client.0.smithi123.stdout: current utilization = 0.00% 2024-03-24T00:39:48.921 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:39:48.921 INFO:tasks.workunit.client.0.smithi123.stdout:checking existing fs: . 2024-03-24T00:39:48.921 INFO:tasks.workunit.client.0.smithi123.stdout:recreating new fileset 2024-03-24T00:39:48.921 INFO:tasks.workunit.client.0.smithi123.stderr:opendir: No such file or directory 2024-03-24T00:39:50.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:50 smithi204 ceph-mon[36345]: pgmap v443: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 128 op/s 2024-03-24T00:39:50.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:50 smithi123 ceph-mon[31487]: pgmap v443: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 128 op/s 2024-03-24T00:39:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:50 smithi177 ceph-mon[36034]: pgmap v443: 97 pgs: 97 active+clean; 6.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 315 KiB/s wr, 128 op/s 2024-03-24T00:39:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:52 smithi204 ceph-mon[36345]: pgmap v444: 97 pgs: 97 active+clean; 28 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-03-24T00:39:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:52 smithi123 ceph-mon[31487]: pgmap v444: 97 pgs: 97 active+clean; 28 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-03-24T00:39:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:52 smithi177 ceph-mon[36034]: pgmap v444: 97 pgs: 97 active+clean; 28 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-03-24T00:39:54.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:54 smithi204 ceph-mon[36345]: pgmap v445: 97 pgs: 97 active+clean; 97 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.0 MiB/s wr, 112 op/s 2024-03-24T00:39:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:54 smithi123 ceph-mon[31487]: pgmap v445: 97 pgs: 97 active+clean; 97 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.0 MiB/s wr, 112 op/s 2024-03-24T00:39:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:54 smithi177 ceph-mon[36034]: pgmap v445: 97 pgs: 97 active+clean; 97 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.0 MiB/s wr, 112 op/s 2024-03-24T00:39:56.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:56 smithi204 ceph-mon[36345]: pgmap v446: 97 pgs: 97 active+clean; 161 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 13 MiB/s wr, 104 op/s 2024-03-24T00:39:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:56 smithi123 ceph-mon[31487]: pgmap v446: 97 pgs: 97 active+clean; 161 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 13 MiB/s wr, 104 op/s 2024-03-24T00:39:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:56 smithi177 ceph-mon[36034]: pgmap v446: 97 pgs: 97 active+clean; 161 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 13 MiB/s wr, 104 op/s 2024-03-24T00:39:58.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:39:58 smithi204 ceph-mon[36345]: pgmap v447: 97 pgs: 97 active+clean; 169 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 14 MiB/s wr, 108 op/s 2024-03-24T00:39:58.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:39:58 smithi123 ceph-mon[31487]: pgmap v447: 97 pgs: 97 active+clean; 169 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 14 MiB/s wr, 108 op/s 2024-03-24T00:39:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:39:58 smithi177 ceph-mon[36034]: pgmap v447: 97 pgs: 97 active+clean; 169 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 14 MiB/s wr, 108 op/s 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 -- 172.21.15.123:0/3687234585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c103280 msgr2=0x7f528c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 --2- 172.21.15.123:0/3687234585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c103280 0x7f528c109b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5288009be0 tx=0x7f528802bef0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 -- 172.21.15.123:0/3687234585 shutdown_connections 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 --2- 172.21.15.123:0/3687234585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c103280 0x7f528c109b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 --2- 172.21.15.123:0/3687234585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c1028c0 0x7f528c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 --2- 172.21.15.123:0/3687234585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 -- 172.21.15.123:0/3687234585 >> 172.21.15.123:0/3687234585 conn(0x7f528c0fcc60 msgr2=0x7f528c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 -- 172.21.15.123:0/3687234585 shutdown_connections 2024-03-24T00:39:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.723+0000 7f5294800640 1 -- 172.21.15.123:0/3687234585 wait complete. 2024-03-24T00:39:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 Processor -- start 2024-03-24T00:39:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 -- start start 2024-03-24T00:39:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 0x7f528c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c103280 0x7f528c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c10d380 con 0x7f528c103280 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c10d200 con 0x7f528c1028c0 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5294800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c10d500 con 0x7f528c1016d0 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.724+0000 7f5291e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:36584/0 (socket says 172.21.15.123:36584) 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 -- 172.21.15.123:0/683667271 learned_addr learned my addr 172.21.15.123:0/683667271 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291400640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 0x7f528c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5292800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c103280 0x7f528c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 -- 172.21.15.123:0/683667271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 msgr2=0x7f528c197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 0x7f528c197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 -- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c103280 msgr2=0x7f528c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c103280 0x7f528c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 -- 172.21.15.123:0/683667271 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c19f4c0 con 0x7f528c1016d0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291400640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 0x7f528c197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5291e00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f528000e9a0 tx=0x7f528000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5286a00640 1 -- 172.21.15.123:0/683667271 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000cd80 con 0x7f528c1016d0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c19f7b0 con 0x7f528c1016d0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5286a00640 1 -- 172.21.15.123:0/683667271 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5280010910 con 0x7f528c1016d0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.725+0000 7f5286a00640 1 -- 172.21.15.123:0/683667271 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280018a40 con 0x7f528c1016d0 2024-03-24T00:39:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.726+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c19fd00 con 0x7f528c1016d0 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.729+0000 7f529743e740 1 Processor -- start 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 -- start start 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f5286a00640 1 -- 172.21.15.123:0/683667271 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5280010ac0 con 0x7f528c1016d0 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb26597ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb26598150 0x55cb2659a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb265513b0 con 0x55cb26592fe0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb264f3830 con 0x55cb265925f0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f529743e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb2649c5d0 con 0x55cb26598150 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f5292800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f5292800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44164/0 (socket says 172.21.15.123:44164) 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f5292800640 1 -- 172.21.15.123:0/677008438 learned_addr learned my addr 172.21.15.123:0/677008438 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.730+0000 7f5291400640 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb26598150 0x55cb2659a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5291e00640 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb26597ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 -- 172.21.15.123:0/677008438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb26598150 msgr2=0x55cb2659a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb26598150 0x55cb2659a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 -- 172.21.15.123:0/677008438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 msgr2=0x55cb26597ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb26597ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 -- 172.21.15.123:0/677008438 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb264b42c0 con 0x55cb265925f0 2024-03-24T00:39:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f5292800640 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f528802c500 tx=0x7f5288009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d747377bac038f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f527b000640 1 -- 172.21.15.123:0/677008438 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288043030 con 0x55cb265925f0 2024-03-24T00:39:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f527b000640 1 -- 172.21.15.123:0/677008438 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5288033580 con 0x55cb265925f0 2024-03-24T00:39:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.731+0000 7f527b000640 1 -- 172.21.15.123:0/677008438 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528802c890 con 0x55cb265925f0 2024-03-24T00:39:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f5286a00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 0x7f52740a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f5286a00640 1 -- 172.21.15.123:0/683667271 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f52800cedc0 con 0x7f528c1016d0 2024-03-24T00:39:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f5291e00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 0x7f52740a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 -- 172.21.15.123:0/677008438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 msgr2=0x55cb26592a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f528802c500 tx=0x7f5288009330 comp rx=0 tx=0).stop 2024-03-24T00:39:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 -- 172.21.15.123:0/677008438 shutdown_connections 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb26598150 0x55cb2659a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb26597ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 --2- 172.21.15.123:0/677008438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb265925f0 0x55cb26592a10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.732+0000 7f529743e740 1 -- 172.21.15.123:0/677008438 >> 172.21.15.123:0/677008438 conn(0x55cb26589750 msgr2=0x55cb26589b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f5291e00640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 0x7f52740a4d60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f528009b410 tx=0x7f52800d2330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 -- 172.21.15.123:0/677008438 shutdown_connections 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 -- 172.21.15.123:0/677008438 wait complete. 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 -- start start 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 Processor -- start 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2659c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.733+0000 7f529743e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f529743e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f529743e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb264afa20 con 0x55cb26592fe0 2024-03-24T00:39:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f529743e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb264ef8d0 con 0x55cb26598150 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f529743e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb2659b040 con 0x55cb265925f0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f529743e740 10 client.0 _ll_register_callbacks cb 0x55cb264ec1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5292800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2659c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40988/0 (socket says 172.21.15.123:40988) 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 -- 172.21.15.123:0/2888948907 learned_addr learned my addr 172.21.15.123:0/2888948907 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5292800640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 -- 172.21.15.123:0/2888948907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 msgr2=0x55cb2659c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2659c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 -- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 msgr2=0x55cb265a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.734+0000 7f5291400640 1 -- 172.21.15.123:0/2888948907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb26208210 con 0x55cb26592fe0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f5291400640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f527c012b10 tx=0x7f527c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b28b6d50f5b958 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f526f400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c00fae0 con 0x55cb26592fe0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f527c00fc80 con 0x55cb26592fe0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5288033580 con 0x55cb26592fe0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f527c004050 con 0x55cb26592fe0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f529743e740 10 client.15531 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.735+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cb262d4620 con 0x55cb26592fe0 2024-03-24T00:39:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5288033580 con 0x55cb26592fe0 2024-03-24T00:39:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f527c010430 con 0x55cb26592fe0 2024-03-24T00:39:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 10 client.15531 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 20 client.15531 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb265a8630 0x55cb265aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 4 client.15531 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb26500d70 con 0x55cb265a8630 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.736+0000 7f5292800640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb265a8630 0x55cb265aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.737+0000 7f526f400640 10 client.15531 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.737+0000 7f5292800640 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb265a8630 0x55cb265aaa50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bef7176da4878c5 server_cookie=3c6e80c503d6c93b in_seq=0 out_seq=0 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f526f400640 1 -- 172.21.15.123:0/2888948907 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cb26500d70 con 0x55cb265a8630 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f526f400640 10 client.15531 handle_command_reply: tid=0 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 client.15531 shutdown 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 msgr2=0x55cb2659cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f527c012b10 tx=0x7f527c00f3c0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 shutdown_connections 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb2659cbe0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb265a8630 0x55cb265aaa50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 --2- 172.21.15.123:0/2888948907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2659c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 >> 172.21.15.123:0/2888948907 conn(0x55cb26589750 msgr2=0x55cb26588950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f526f400640 0 client.15531 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 shutdown_connections 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.738+0000 7f529743e740 1 -- 172.21.15.123:0/2888948907 wait complete. 2024-03-24T00:39:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.739+0000 7f529743e740 20 client.15531 trim_cache size 0 max 16384 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 Processor -- start 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 -- start start 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a4800 con 0x55cb26592fe0 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a6dc0 con 0x55cb26598150 2024-03-24T00:39:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f529743e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a4e00 con 0x55cb265925f0 2024-03-24T00:39:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.753+0000 7f5292800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40998/0 (socket says 172.21.15.123:40998) 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5292800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:36608/0 (socket says 172.21.15.123:36608) 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 -- 172.21.15.123:0/3556883180 learned_addr learned my addr 172.21.15.123:0/3556883180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5292800640 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 -- 172.21.15.123:0/3556883180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 msgr2=0x55cb2658d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 -- 172.21.15.123:0/3556883180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 msgr2=0x55cb265843a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265843a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 -- 172.21.15.123:0/3556883180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb2658a4c0 con 0x55cb26592fe0 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5292800640 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f5291400640 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f527c010820 tx=0x7f527c010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba5acc58550d5da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f527a600640 1 -- 172.21.15.123:0/3556883180 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c010430 con 0x55cb26592fe0 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.754+0000 7f527a600640 1 -- 172.21.15.123:0/3556883180 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f527c0105d0 con 0x55cb26592fe0 2024-03-24T00:39:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f527a600640 1 -- 172.21.15.123:0/3556883180 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c0039f0 con 0x55cb26592fe0 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 -- 172.21.15.123:0/3556883180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 msgr2=0x55cb265a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f527c010820 tx=0x7f527c010ca0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 -- 172.21.15.123:0/3556883180 shutdown_connections 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265843a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a3be0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 --2- 172.21.15.123:0/3556883180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb2658d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 -- 172.21.15.123:0/3556883180 >> 172.21.15.123:0/3556883180 conn(0x55cb26589750 msgr2=0x55cb2658c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 -- 172.21.15.123:0/3556883180 shutdown_connections 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.755+0000 7f529743e740 1 -- 172.21.15.123:0/3556883180 wait complete. 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 -- start start 2024-03-24T00:39:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 Processor -- start 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb265a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a5440 con 0x55cb26592fe0 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a6760 con 0x55cb26598150 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1a5aa0 con 0x55cb265925f0 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.756+0000 7f529743e740 10 client.0 _ll_register_callbacks cb 0x7f527c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41000/0 (socket says 172.21.15.123:41000) 2024-03-24T00:39:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 -- 172.21.15.123:0/2457385331 learned_addr learned my addr 172.21.15.123:0/2457385331 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5292800640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5292800640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb265a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 -- 172.21.15.123:0/2457385331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 msgr2=0x55cb265a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb265a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 -- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 msgr2=0x55cb265a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 -- 172.21.15.123:0/2457385331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb265a28f0 con 0x55cb26592fe0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f5291400640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f527c00ff40 tx=0x7f527c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe12e8891eb1526 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f526f400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c010430 con 0x55cb26592fe0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f527c005270 con 0x55cb26592fe0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.757+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f527c0105d0 con 0x55cb26592fe0 2024-03-24T00:39:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f527c010a40 con 0x55cb26592fe0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 10 client.15537 fetch_fsmap learned FSMap version 19 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cb26604c70 con 0x55cb26592fe0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f527c0105d0 con 0x55cb26592fe0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f527c02e020 con 0x55cb26592fe0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 10 client.15537 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 10 client.15537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 20 client.15537 populate_metadata read hostname 'smithi123' 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb266054b0 0x55cb266078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 4 client.15537 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.758+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cb26584a80 con 0x55cb266054b0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.759+0000 7f5292800640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb266054b0 0x55cb266078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.759+0000 7f5292800640 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb266054b0 0x55cb266078d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f0fa967fe379193 server_cookie=8c431be162accf1a in_seq=0 out_seq=0 2024-03-24T00:39:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.759+0000 7f526f400640 10 client.15537 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f526f400640 1 -- 172.21.15.123:0/2457385331 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cb26584a80 con 0x55cb266054b0 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f526f400640 10 client.15537 handle_command_reply: tid=0 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 client.15537 shutdown 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 msgr2=0x55cb265a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f527c00ff40 tx=0x7f527c005f90 comp rx=0 tx=0).stop 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 shutdown_connections 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb26598150 0x55cb265a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb26592fe0 0x55cb265a1e40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb266054b0 0x55cb266078d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 --2- 172.21.15.123:0/2457385331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb265925f0 0x55cb265a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f526f400640 0 client.15537 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 >> 172.21.15.123:0/2457385331 conn(0x55cb26589750 msgr2=0x55cb265880c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 shutdown_connections 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 1 -- 172.21.15.123:0/2457385331 wait complete. 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.792+0000 7f529743e740 20 client.15537 trim_cache size 0 max 16384 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:39:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 msgr2=0x7f52740a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 0x7f52740a4d60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f528009b410 tx=0x7f52800d2330 comp rx=0 tx=0).stop 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 msgr2=0x7f528c1978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f528000e9a0 tx=0x7f528000ee60 comp rx=0 tx=0).stop 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 shutdown_connections 2024-03-24T00:39:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f52740a28a0 0x7f52740a4d60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f528c103280 0x7f528c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f528c1028c0 0x7f528c197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 --2- 172.21.15.123:0/683667271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f528c1016d0 0x7f528c1978a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 >> 172.21.15.123:0/683667271 conn(0x7f528c0fcc60 msgr2=0x7f528c105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 shutdown_connections 2024-03-24T00:39:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:39:58.793+0000 7f5294800640 1 -- 172.21.15.123:0/683667271 wait complete. 2024-03-24T00:40:00.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:00 smithi123 ceph-mon[31487]: pgmap v448: 97 pgs: 97 active+clean; 309 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 26 MiB/s wr, 87 op/s 2024-03-24T00:40:00.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:00 smithi123 ceph-mon[31487]: overall HEALTH_OK 2024-03-24T00:40:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:00 smithi177 ceph-mon[36034]: pgmap v448: 97 pgs: 97 active+clean; 309 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 26 MiB/s wr, 87 op/s 2024-03-24T00:40:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:00 smithi177 ceph-mon[36034]: overall HEALTH_OK 2024-03-24T00:40:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:00 smithi204 ceph-mon[36345]: pgmap v448: 97 pgs: 97 active+clean; 309 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 26 MiB/s wr, 87 op/s 2024-03-24T00:40:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:00 smithi204 ceph-mon[36345]: overall HEALTH_OK 2024-03-24T00:40:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:01 smithi177 ceph-mon[36034]: pgmap v449: 97 pgs: 97 active+clean; 890 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 74 MiB/s wr, 103 op/s 2024-03-24T00:40:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:01 smithi204 ceph-mon[36345]: pgmap v449: 97 pgs: 97 active+clean; 890 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 74 MiB/s wr, 103 op/s 2024-03-24T00:40:01.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:01 smithi123 ceph-mon[31487]: pgmap v449: 97 pgs: 97 active+clean; 890 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 74 MiB/s wr, 103 op/s 2024-03-24T00:40:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:04 smithi204 ceph-mon[36345]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 126 MiB/s wr, 95 op/s 2024-03-24T00:40:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:04 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:04.622 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:04 smithi123 ceph-mon[31487]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 126 MiB/s wr, 95 op/s 2024-03-24T00:40:04.622 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:04 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:04 smithi177 ceph-mon[36034]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 126 MiB/s wr, 95 op/s 2024-03-24T00:40:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:04 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:06 smithi204 ceph-mon[36345]: pgmap v451: 97 pgs: 97 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 196 MiB/s wr, 102 op/s 2024-03-24T00:40:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:06.572 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:40:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:06 smithi123 ceph-mon[31487]: pgmap v451: 97 pgs: 97 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 196 MiB/s wr, 102 op/s 2024-03-24T00:40:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:40:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:06 smithi177 ceph-mon[36034]: pgmap v451: 97 pgs: 97 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 196 MiB/s wr, 102 op/s 2024-03-24T00:40:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:40:08.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:40:08.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:40:08.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 -- 172.21.15.123:0/2503333209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a01086a0 msgr2=0x7f90a0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 --2- 172.21.15.123:0/2503333209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a01086a0 0x7f90a0108ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f908c009900 tx=0x7f908c02bc80 comp rx=0 tx=0).stop 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 -- 172.21.15.123:0/2503333209 shutdown_connections 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 --2- 172.21.15.123:0/2503333209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a010ff20 0x7f90a0112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 --2- 172.21.15.123:0/2503333209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a010f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 --2- 172.21.15.123:0/2503333209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a01086a0 0x7f90a0108ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.469+0000 7f90a5400640 1 -- 172.21.15.123:0/2503333209 >> 172.21.15.123:0/2503333209 conn(0x7f90a006e500 msgr2=0x7f90a0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.470+0000 7f90a5400640 1 -- 172.21.15.123:0/2503333209 shutdown_connections 2024-03-24T00:40:08.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.470+0000 7f90a5400640 1 -- 172.21.15.123:0/2503333209 wait complete. 2024-03-24T00:40:08.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.475+0000 7f90a5400640 1 Processor -- start 2024-03-24T00:40:08.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.475+0000 7f90a5400640 1 -- start start 2024-03-24T00:40:08.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.475+0000 7f90a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a01086a0 0x7f90a01a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.475+0000 7f90a5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.475+0000 7f90a5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a010ff20 0x7f90a01a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.476+0000 7f90a5400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0115c90 con 0x7f90a0109000 2024-03-24T00:40:08.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.476+0000 7f90a5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0115b10 con 0x7f90a01086a0 2024-03-24T00:40:08.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.476+0000 7f90a5400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0115e10 con 0x7f90a010ff20 2024-03-24T00:40:08.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.477+0000 7f909e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a01086a0 0x7f90a01a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.477+0000 7f909de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.477+0000 7f909de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35912/0 (socket says 172.21.15.123:35912) 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.477+0000 7f909de00640 1 -- 172.21.15.123:0/2180846864 learned_addr learned my addr 172.21.15.123:0/2180846864 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.478+0000 7f909de00640 1 -- 172.21.15.123:0/2180846864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a010ff20 msgr2=0x7f90a01a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.478+0000 7f909de00640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a010ff20 0x7f90a01a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.478+0000 7f909de00640 1 -- 172.21.15.123:0/2180846864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a01086a0 msgr2=0x7f90a01a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.478+0000 7f909de00640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a01086a0 0x7f90a01a25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.478+0000 7f909de00640 1 -- 172.21.15.123:0/2180846864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a01aa1c0 con 0x7f90a0109000 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.482+0000 7f909de00640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f909000b450 tx=0x7f909000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.482+0000 7f9097400640 1 -- 172.21.15.123:0/2180846864 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090021070 con 0x7f90a0109000 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.482+0000 7f9097400640 1 -- 172.21.15.123:0/2180846864 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f909001c920 con 0x7f90a0109000 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.482+0000 7f9097400640 1 -- 172.21.15.123:0/2180846864 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909001a9c0 con 0x7f90a0109000 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.482+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a010ba20 con 0x7f90a0109000 2024-03-24T00:40:08.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.483+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a010bf20 con 0x7f90a0109000 2024-03-24T00:40:08.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.487+0000 7f90a805b740 1 Processor -- start 2024-03-24T00:40:08.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.487+0000 7f90a805b740 1 -- start start 2024-03-24T00:40:08.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.488+0000 7f90a805b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d802a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f90a805b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f90a805b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d859e0 0x55d4d8d87e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f90a805b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8ce5250 con 0x55d4d8d7fe80 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f90a805b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8c8da30 con 0x55d4d8d859e0 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f90a805b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8ce12f0 con 0x55d4d8d80870 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.489+0000 7f9097400640 1 -- 172.21.15.123:0/2180846864 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f909001cad0 con 0x7f90a0109000 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.490+0000 7f909e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.490+0000 7f909e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:41836/0 (socket says 172.21.15.123:41836) 2024-03-24T00:40:08.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.490+0000 7f909e800640 1 -- 172.21.15.123:0/2958901161 learned_addr learned my addr 172.21.15.123:0/2958901161 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.491+0000 7f909de00640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d859e0 0x55d4d8d87e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909e800640 1 -- 172.21.15.123:0/2958901161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d859e0 msgr2=0x55d4d8d87e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909f200640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d802a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909e800640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d859e0 0x55d4d8d87e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909e800640 1 -- 172.21.15.123:0/2958901161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 msgr2=0x55d4d8d802a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909e800640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d802a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909e800640 1 -- 172.21.15.123:0/2958901161 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d8c78690 con 0x55d4d8d80870 2024-03-24T00:40:08.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.493+0000 7f909f200640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:08.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.494+0000 7f9097400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 0x7f90840b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.494+0000 7f9097400640 1 -- 172.21.15.123:0/2180846864 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f90900ce350 con 0x7f90a0109000 2024-03-24T00:40:08.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.494+0000 7f909f200640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 0x7f90840b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.497+0000 7f909e800640 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f908c036040 tx=0x7f908c02c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc5d141a94346f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.497+0000 7f908b400640 1 -- 172.21.15.123:0/2958901161 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c042070 con 0x55d4d8d80870 2024-03-24T00:40:08.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.498+0000 7f908b400640 1 -- 172.21.15.123:0/2958901161 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f908c005000 con 0x55d4d8d80870 2024-03-24T00:40:08.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.498+0000 7f908b400640 1 -- 172.21.15.123:0/2958901161 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c004a20 con 0x55d4d8d80870 2024-03-24T00:40:08.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.499+0000 7f909f200640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 0x7f90840b5e20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55d4d8d810d0 tx=0x7f9098007340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 -- 172.21.15.123:0/2958901161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 msgr2=0x55d4d8d85370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f908c036040 tx=0x7f908c02c9d0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 -- 172.21.15.123:0/2958901161 shutdown_connections 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d859e0 0x55d4d8d87e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d85370 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 --2- 172.21.15.123:0/2958901161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.500+0000 7f90a805b740 1 -- 172.21.15.123:0/2958901161 >> 172.21.15.123:0/2958901161 conn(0x55d4d8d48200 msgr2=0x55d4d8d7b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.501+0000 7f90a805b740 1 -- 172.21.15.123:0/2958901161 shutdown_connections 2024-03-24T00:40:08.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.501+0000 7f90a805b740 1 -- 172.21.15.123:0/2958901161 wait complete. 2024-03-24T00:40:08.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.502+0000 7f90a805b740 1 -- start start 2024-03-24T00:40:08.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.502+0000 7f90a805b740 1 Processor -- start 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d89e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d8a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d859e0 0x55d4d8d913a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8d88930 con 0x55d4d8d859e0 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8ca0e80 con 0x55d4d8d7fe80 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d8d88ab0 con 0x55d4d8d80870 2024-03-24T00:40:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.503+0000 7f90a805b740 10 client.0 _ll_register_callbacks cb 0x55d4d8cb3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d8a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d8a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:41842/0 (socket says 172.21.15.123:41842) 2024-03-24T00:40:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d89e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909de00640 1 -- 172.21.15.123:0/119873209 learned_addr learned my addr 172.21.15.123:0/119873209 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909f200640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d859e0 0x55d4d8d913a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 -- 172.21.15.123:0/119873209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 msgr2=0x55d4d8d8a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d8a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 -- 172.21.15.123:0/119873209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d859e0 msgr2=0x55d4d8d913a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d859e0 0x55d4d8d913a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.504+0000 7f909e800640 1 -- 172.21.15.123:0/119873209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4d8ac37d0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f909e800640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d89e20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55d4d8d7c740 tx=0x7f908c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d290e9a7092a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f907fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c042030 con 0x55d4d8d7fe80 2024-03-24T00:40:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f908c0049b0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f908c005000 con 0x55d4d8d7fe80 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f908c004bb0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f90a805b740 10 client.25649 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.505+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d4d89f93f0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f908c005000 con 0x55d4d8d7fe80 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f908c034440 con 0x55d4d8d7fe80 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 10 client.25649 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 10 client.25649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 20 client.25649 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8d95800 0x55d4d8d97bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 4 client.25649 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4d8cf2790 con 0x55d4d8d95800 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.506+0000 7f909de00640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8d95800 0x55d4d8d97bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.507+0000 7f909de00640 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8d95800 0x55d4d8d97bf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94ac7d543e00dbf9 server_cookie=2893fee1f6360040 in_seq=0 out_seq=0 2024-03-24T00:40:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.507+0000 7f907fe00640 10 client.25649 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:08.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.507+0000 7f907fe00640 1 -- 172.21.15.123:0/119873209 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d4d8cf2790 con 0x55d4d8d95800 2024-03-24T00:40:08.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.507+0000 7f907fe00640 10 client.25649 handle_command_reply: tid=0 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.507+0000 7f90a805b740 1 client.25649 shutdown 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 msgr2=0x55d4d8d89e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d89e20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55d4d8d7c740 tx=0x7f908c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 shutdown_connections 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d859e0 0x55d4d8d913a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8d95800 0x55d4d8d97bf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d80870 0x55d4d8d8a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 --2- 172.21.15.123:0/119873209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d89e20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f907fe00640 0 client.25649 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 >> 172.21.15.123:0/119873209 conn(0x55d4d8d48200 msgr2=0x55d4d8d7aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 shutdown_connections 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 1 -- 172.21.15.123:0/119873209 wait complete. 2024-03-24T00:40:08.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.508+0000 7f90a805b740 20 client.25649 trim_cache size 0 max 16384 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 Processor -- start 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 -- start start 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d81b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d7a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01ad1a0 con 0x55d4d8d80870 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01af760 con 0x55d4d8d7fe80 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f90a805b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01ad7a0 con 0x55d4d8d859e0 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.528+0000 7f909de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d81b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35938/0 (socket says 172.21.15.123:35938) 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909de00640 1 -- 172.21.15.123:0/4050429843 learned_addr learned my addr 172.21.15.123:0/4050429843 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909f200640 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d7a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 -- 172.21.15.123:0/4050429843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 msgr2=0x55d4d8d7a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d7a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 -- 172.21.15.123:0/4050429843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 msgr2=0x55d4d8d91360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 -- 172.21.15.123:0/4050429843 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d89b90f0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909de00640 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f909e800640 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d81b70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f908c0348e0 tx=0x7f908c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a79c39c364d669d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.529+0000 7f908aa00640 1 -- 172.21.15.123:0/4050429843 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c042070 con 0x55d4d8d7fe80 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f908aa00640 1 -- 172.21.15.123:0/4050429843 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f908c005000 con 0x55d4d8d7fe80 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f908aa00640 1 -- 172.21.15.123:0/4050429843 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c034440 con 0x55d4d8d7fe80 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 -- 172.21.15.123:0/4050429843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 msgr2=0x55d4d8d81b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d81b70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f908c0348e0 tx=0x7f908c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 -- 172.21.15.123:0/4050429843 shutdown_connections 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d7a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 --2- 172.21.15.123:0/4050429843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d81b70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 -- 172.21.15.123:0/4050429843 >> 172.21.15.123:0/4050429843 conn(0x55d4d8d48200 msgr2=0x55d4d8d7a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 -- 172.21.15.123:0/4050429843 shutdown_connections 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.530+0000 7f90a805b740 1 -- 172.21.15.123:0/4050429843 wait complete. 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 -- start start 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 Processor -- start 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d8eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d8fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01adde0 con 0x55d4d8d80870 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01af100 con 0x55d4d8d7fe80 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01ae440 con 0x55d4d8d859e0 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f90a805b740 10 client.0 _ll_register_callbacks cb 0x7f908c0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.531+0000 7f909e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56412/0 (socket says 172.21.15.123:56412) 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 -- 172.21.15.123:0/985067283 learned_addr learned my addr 172.21.15.123:0/985067283 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909f200640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d8fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909de00640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d8eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 -- 172.21.15.123:0/985067283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 msgr2=0x55d4d8d8fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d8fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 -- 172.21.15.123:0/985067283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 msgr2=0x55d4d8d8eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d8eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 -- 172.21.15.123:0/985067283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4d8d900a0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f909e800640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f908c009850 tx=0x7f908c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4416c079534dd59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f907fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c042070 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f908c005000 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.532+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f908c047450 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f908c047690 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 10 client.25655 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d4d8d913b0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f908c005000 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f908c03e9f0 con 0x55d4d8d7fe80 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 10 client.25655 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 10 client.25655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 20 client.25655 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8df0ff0 0x55d4d8df33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 4 client.25655 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:40:08.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.533+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d4d8cecae0 con 0x55d4d8df0ff0 2024-03-24T00:40:08.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f909de00640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8df0ff0 0x55d4d8df33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f907fe00640 10 client.25655 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f909de00640 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8df0ff0 0x55d4d8df33e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c76af318df532b9f server_cookie=7a1cbf70024a3e1 in_seq=0 out_seq=0 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f907fe00640 1 -- 172.21.15.123:0/985067283 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d4d8cecae0 con 0x55d4d8df0ff0 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f907fe00640 10 client.25655 handle_command_reply: tid=0 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f90a805b740 1 client.25655 shutdown 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 msgr2=0x55d4d8d8e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.534+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f908c009850 tx=0x7f908c004730 comp rx=0 tx=0).stop 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 shutdown_connections 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d4d8df0ff0 0x55d4d8df33e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d4d8d859e0 0x55d4d8d8fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d4d8d80870 0x55d4d8d8eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 --2- 172.21.15.123:0/985067283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d8d7fe80 0x55d4d8d8e560 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 >> 172.21.15.123:0/985067283 conn(0x55d4d8d48200 msgr2=0x55d4d8d7a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f907fe00640 0 client.25655 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 shutdown_connections 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 1 -- 172.21.15.123:0/985067283 wait complete. 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a805b740 20 client.25655 trim_cache size 0 max 16384 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:40:08.539 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 msgr2=0x7f90840b5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 0x7f90840b5e20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55d4d8d810d0 tx=0x7f9098007340 comp rx=0 tx=0).stop 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 msgr2=0x7f90a01a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.535+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f909000b450 tx=0x7f909000b910 comp rx=0 tx=0).stop 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 shutdown_connections 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90a010ff20 0x7f90a01a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f90840b3960 0x7f90840b5e20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a0109000 0x7f90a01a2ae0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 --2- 172.21.15.123:0/2180846864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90a01086a0 0x7f90a01a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 >> 172.21.15.123:0/2180846864 conn(0x7f90a006e500 msgr2=0x7f90a00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 shutdown_connections 2024-03-24T00:40:08.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.536+0000 7f90a5400640 1 -- 172.21.15.123:0/2180846864 wait complete. 2024-03-24T00:40:08.549 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:08 smithi123 ceph-mon[31487]: pgmap v452: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 212 MiB/s wr, 107 op/s 2024-03-24T00:40:08.552 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:40:08.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:08 smithi204 ceph-mon[36345]: pgmap v452: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 212 MiB/s wr, 107 op/s 2024-03-24T00:40:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:08 smithi177 ceph-mon[36034]: pgmap v452: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 212 MiB/s wr, 107 op/s 2024-03-24T00:40:08.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.773+0000 7f6542a00640 1 -- 172.21.15.123:0/1438389928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 msgr2=0x7f653c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.773+0000 7f6542a00640 1 --2- 172.21.15.123:0/1438389928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c106e90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6524009900 tx=0x7f652402bc30 comp rx=0 tx=0).stop 2024-03-24T00:40:08.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 -- 172.21.15.123:0/1438389928 shutdown_connections 2024-03-24T00:40:08.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 --2- 172.21.15.123:0/1438389928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 0x7f653c1097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 --2- 172.21.15.123:0/1438389928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c106e90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 --2- 172.21.15.123:0/1438389928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c0ff6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 -- 172.21.15.123:0/1438389928 >> 172.21.15.123:0/1438389928 conn(0x7f653c0faea0 msgr2=0x7f653c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:08.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 -- 172.21.15.123:0/1438389928 shutdown_connections 2024-03-24T00:40:08.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.774+0000 7f6542a00640 1 -- 172.21.15.123:0/1438389928 wait complete. 2024-03-24T00:40:08.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.776+0000 7f6542a00640 1 Processor -- start 2024-03-24T00:40:08.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.776+0000 7f6542a00640 1 -- start start 2024-03-24T00:40:08.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 0x7f653c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10d370 con 0x7f653c0ff2c0 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10d1f0 con 0x7f653c0ffcb0 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.777+0000 7f6542a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10d4f0 con 0x7f653c1073d0 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35972/0 (socket says 172.21.15.123:35972) 2024-03-24T00:40:08.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 -- 172.21.15.123:0/1501721130 learned_addr learned my addr 172.21.15.123:0/1501721130 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f6540a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 0x7f653c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 -- 172.21.15.123:0/1501721130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 msgr2=0x7f653c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 0x7f653c1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f653be00640 1 -- 172.21.15.123:0/1501721130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 msgr2=0x7f653c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.778+0000 7f6533e00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.779+0000 7f653be00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.779+0000 7f653be00640 1 -- 172.21.15.123:0/1501721130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c1a17a0 con 0x7f653c0ff2c0 2024-03-24T00:40:08.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.779+0000 7f6533e00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:08.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.779+0000 7f653be00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f652c00da90 tx=0x7f652c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.780+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c017070 con 0x7f653c0ff2c0 2024-03-24T00:40:08.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.780+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c1a1a90 con 0x7f653c0ff2c0 2024-03-24T00:40:08.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.780+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c1a2030 con 0x7f653c0ff2c0 2024-03-24T00:40:08.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.781+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f652c009c00 con 0x7f653c0ff2c0 2024-03-24T00:40:08.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.781+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c020060 con 0x7f653c0ff2c0 2024-03-24T00:40:08.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.784+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180052c0 con 0x7f653c0ff2c0 2024-03-24T00:40:08.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.787+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f652c00f620 con 0x7f653c0ff2c0 2024-03-24T00:40:08.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.789+0000 7f6539600640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 0x7f65200a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:08.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.789+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f652c0ce7d0 con 0x7f653c0ff2c0 2024-03-24T00:40:08.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.790+0000 7f6533e00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 0x7f65200a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:08.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.792+0000 7f6533e00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 0x7f65200a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f65240096c0 tx=0x7f652402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:08.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:08.798+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f652c09b820 con 0x7f653c0ff2c0 2024-03-24T00:40:09.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.184+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65180055b0 con 0x7f653c0ff2c0 2024-03-24T00:40:09.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.185+0000 7f6539600640 1 -- 172.21.15.123:0/1501721130 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f652c09b120 con 0x7f653c0ff2c0 2024-03-24T00:40:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:40:09.185 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:40:09.185 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:40:09.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 msgr2=0x7f65200a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 0x7f65200a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f65240096c0 tx=0x7f652402df90 comp rx=0 tx=0).stop 2024-03-24T00:40:09.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 msgr2=0x7f653c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f652c00da90 tx=0x7f652c00df50 comp rx=0 tx=0).stop 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 shutdown_connections 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f653c1073d0 0x7f653c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f653c0ffcb0 0x7f653c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f65200a2900 0x7f65200a4dc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.187+0000 7f6542a00640 1 --2- 172.21.15.123:0/1501721130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f653c0ff2c0 0x7f653c199bd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.188+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 >> 172.21.15.123:0/1501721130 conn(0x7f653c0faea0 msgr2=0x7f653c0fb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.188+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 shutdown_connections 2024-03-24T00:40:09.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.188+0000 7f6542a00640 1 -- 172.21.15.123:0/1501721130 wait complete. 2024-03-24T00:40:09.207 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:40:09.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:40:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 -- 172.21.15.123:0/795495713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 msgr2=0x7fc0a0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/795495713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0101ad0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc098009ca0 tx=0x7fc09802c100 comp rx=0 tx=0).stop 2024-03-24T00:40:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 -- 172.21.15.123:0/795495713 shutdown_connections 2024-03-24T00:40:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/795495713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a0103260 0x7fc0a0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/795495713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a01028a0 0x7fc0a0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/795495713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0101ad0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 -- 172.21.15.123:0/795495713 >> 172.21.15.123:0/795495713 conn(0x7fc0a00fcc40 msgr2=0x7fc0a00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 -- 172.21.15.123:0/795495713 shutdown_connections 2024-03-24T00:40:09.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.467+0000 7fc0a4e00640 1 -- 172.21.15.123:0/795495713 wait complete. 2024-03-24T00:40:09.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 Processor -- start 2024-03-24T00:40:09.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 -- start start 2024-03-24T00:40:09.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a0103260 0x7fc0a01a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a010d380 con 0x7fc0a01016b0 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a010d200 con 0x7fc0a01028a0 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.468+0000 7fc0a4e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a010d500 con 0x7fc0a0103260 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56468/0 (socket says 172.21.15.123:56468) 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 -- 172.21.15.123:0/2277538388 learned_addr learned my addr 172.21.15.123:0/2277538388 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09de00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09e800640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a0103260 0x7fc0a01a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 -- 172.21.15.123:0/2277538388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a0103260 msgr2=0x7fc0a01a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a0103260 0x7fc0a01a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 -- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 msgr2=0x7fc0a0199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.469+0000 7fc09d400640 1 -- 172.21.15.123:0/2277538388 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a01a1900 con 0x7fc0a01028a0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc09d400640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00e9f0 tx=0x7fc08c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc096a00640 1 -- 172.21.15.123:0/2277538388 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00cd80 con 0x7fc0a01028a0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc096a00640 1 -- 172.21.15.123:0/2277538388 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc08c010910 con 0x7fc0a01028a0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a01a1bf0 con 0x7fc0a01028a0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc096a00640 1 -- 172.21.15.123:0/2277538388 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c018990 con 0x7fc0a01028a0 2024-03-24T00:40:09.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.470+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a01a2110 con 0x7fc0a01028a0 2024-03-24T00:40:09.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.473+0000 7fc0a7a78740 1 Processor -- start 2024-03-24T00:40:09.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.473+0000 7fc0a7a78740 1 -- start start 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b85810 0x5576e9b85c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9b8dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9aea2c0 con 0x5576e9b85810 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9a92c90 con 0x5576e9b873a0 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9ae6360 con 0x5576e9b869e0 2024-03-24T00:40:09.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc09de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc09de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:41920/0 (socket says 172.21.15.123:41920) 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc09de00640 1 -- 172.21.15.123:0/2949107798 learned_addr learned my addr 172.21.15.123:0/2949107798 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc09de00640 1 -- 172.21.15.123:0/2949107798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 msgr2=0x5576e9b8dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.474+0000 7fc09de00640 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9b8dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.475+0000 7fc09de00640 1 -- 172.21.15.123:0/2949107798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b85810 msgr2=0x5576e9b85c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.475+0000 7fc09de00640 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b85810 0x5576e9b85c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.475+0000 7fc09de00640 1 -- 172.21.15.123:0/2949107798 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576e9ab7080 con 0x5576e9b869e0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.475+0000 7fc09de00640 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc09802c710 tx=0x7fc098005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea269887ef3971a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.475+0000 7fc096a00640 1 -- 172.21.15.123:0/2277538388 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc08c010ac0 con 0x7fc0a01028a0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.476+0000 7fc08b400640 1 -- 172.21.15.123:0/2949107798 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098043070 con 0x5576e9b869e0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.476+0000 7fc08b400640 1 -- 172.21.15.123:0/2949107798 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc098003970 con 0x5576e9b869e0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.476+0000 7fc08b400640 1 -- 172.21.15.123:0/2949107798 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098039040 con 0x5576e9b869e0 2024-03-24T00:40:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 -- 172.21.15.123:0/2949107798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 msgr2=0x5576e9b86e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc09802c710 tx=0x7fc098005f80 comp rx=0 tx=0).stop 2024-03-24T00:40:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 -- 172.21.15.123:0/2949107798 shutdown_connections 2024-03-24T00:40:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9b8dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b869e0 0x5576e9b86e60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/2949107798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b85810 0x5576e9b85c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc0a7a78740 1 -- 172.21.15.123:0/2949107798 >> 172.21.15.123:0/2949107798 conn(0x5576e9b80c00 msgr2=0x5576e9b81000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.477+0000 7fc096a00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 0x7fc0840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.478+0000 7fc096a00640 1 -- 172.21.15.123:0/2277538388 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc08c0ceda0 con 0x7fc0a01028a0 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.478+0000 7fc0a7a78740 1 -- 172.21.15.123:0/2949107798 shutdown_connections 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.478+0000 7fc09de00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 0x7fc0840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.478+0000 7fc0a7a78740 1 -- 172.21.15.123:0/2949107798 wait complete. 2024-03-24T00:40:09.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.479+0000 7fc0a7a78740 1 -- start start 2024-03-24T00:40:09.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.479+0000 7fc0a7a78740 1 Processor -- start 2024-03-24T00:40:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.480+0000 7fc09de00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 0x7fc0840a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc098037040 tx=0x7fc098000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b98820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9b96b60 con 0x5576e9b873a0 2024-03-24T00:40:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9aa60d0 con 0x5576e9b869e0 2024-03-24T00:40:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e9b96ce0 con 0x5576e9b85810 2024-03-24T00:40:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc0a7a78740 10 client.0 _ll_register_callbacks cb 0x5576e9aaae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc09e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc09e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36008/0 (socket says 172.21.15.123:36008) 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.481+0000 7fc09e800640 1 -- 172.21.15.123:0/1659669593 learned_addr learned my addr 172.21.15.123:0/1659669593 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09d400640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b98820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09de00640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09e800640 1 -- 172.21.15.123:0/1659669593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 msgr2=0x5576e9b982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09e800640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09e800640 1 -- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 msgr2=0x5576e9b98820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09e800640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b98820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.482+0000 7fc09e800640 1 -- 172.21.15.123:0/1659669593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576e97fe3f0 con 0x5576e9b873a0 2024-03-24T00:40:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.483+0000 7fc09e800640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e450 tx=0x7fc09000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f2ba488581b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.483+0000 7fc07fe00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.483+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090017070 con 0x5576e9b873a0 2024-03-24T00:40:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.483+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc098003970 con 0x5576e9b873a0 2024-03-24T00:40:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.483+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc090013680 con 0x5576e9b873a0 2024-03-24T00:40:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.484+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc090023050 con 0x5576e9b873a0 2024-03-24T00:40:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.484+0000 7fc0a7a78740 10 client.15564 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.484+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5576e98c87d0 con 0x5576e9b873a0 2024-03-24T00:40:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.485+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc098003970 con 0x5576e9b873a0 2024-03-24T00:40:09.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.485+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc0900129d0 con 0x5576e9b873a0 2024-03-24T00:40:09.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.485+0000 7fc0a7a78740 10 client.15564 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:09.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.485+0000 7fc0a7a78740 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:09.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc0a7a78740 20 client.15564 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:09.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9ba3de0 0x5576e9ba6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc0a7a78740 4 client.15564 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576e9af7800 con 0x5576e9ba3de0 2024-03-24T00:40:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc09de00640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9ba3de0 0x5576e9ba6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc09de00640 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9ba3de0 0x5576e9ba6200 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f749d3b442b05f78 server_cookie=50e0fb6281ad6b88 in_seq=0 out_seq=0 2024-03-24T00:40:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.486+0000 7fc07fe00640 10 client.15564 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.487+0000 7fc07fe00640 1 -- 172.21.15.123:0/1659669593 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5576e9af7800 con 0x5576e9ba3de0 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.487+0000 7fc07fe00640 10 client.15564 handle_command_reply: tid=0 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.487+0000 7fc0a7a78740 1 client.15564 shutdown 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.487+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 msgr2=0x5576e9b9f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.487+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e450 tx=0x7fc09000e910 comp rx=0 tx=0).stop 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 shutdown_connections 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9f860 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b98820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9ba3de0 0x5576e9ba6200 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1659669593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 >> 172.21.15.123:0/1659669593 conn(0x5576e9b80c00 msgr2=0x5576e9b7feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc07fe00640 0 client.15564 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:09.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.488+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 shutdown_connections 2024-03-24T00:40:09.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.489+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1659669593 wait complete. 2024-03-24T00:40:09.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.489+0000 7fc0a7a78740 20 client.15564 trim_cache size 0 max 16384 2024-03-24T00:40:09.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.520+0000 7fc0a7a78740 1 Processor -- start 2024-03-24T00:40:09.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.521+0000 7fc0a7a78740 1 -- start start 2024-03-24T00:40:09.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.523+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b84570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b869e0 0x5576e9b9f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a4af0 con 0x5576e9b869e0 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a7090 con 0x5576e9b873a0 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a50f0 con 0x5576e9b85810 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc09e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc09e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56480/0 (socket says 172.21.15.123:56480) 2024-03-24T00:40:09.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc09e800640 1 -- 172.21.15.123:0/573007499 learned_addr learned my addr 172.21.15.123:0/573007499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.524+0000 7fc09de00640 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b84570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09d400640 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b869e0 0x5576e9b9f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09e800640 1 -- 172.21.15.123:0/573007499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 msgr2=0x5576e9b84570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09e800640 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b84570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09e800640 1 -- 172.21.15.123:0/573007499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b869e0 msgr2=0x5576e9b9f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09e800640 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b869e0 0x5576e9b9f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.525+0000 7fc09e800640 1 -- 172.21.15.123:0/573007499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576e9b7c510 con 0x5576e9b873a0 2024-03-24T00:40:09.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.526+0000 7fc09e800640 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc090012660 tx=0x7fc090012690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e87a2e0638e22055 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.526+0000 7fc08aa00640 1 -- 172.21.15.123:0/573007499 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090017070 con 0x5576e9b873a0 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.526+0000 7fc08aa00640 1 -- 172.21.15.123:0/573007499 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc090020030 con 0x5576e9b873a0 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.526+0000 7fc08aa00640 1 -- 172.21.15.123:0/573007499 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09001c6a0 con 0x5576e9b873a0 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.528+0000 7fc0a7a78740 1 -- 172.21.15.123:0/573007499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 msgr2=0x5576e9baaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.528+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc090012660 tx=0x7fc090012690 comp rx=0 tx=0).stop 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 -- 172.21.15.123:0/573007499 shutdown_connections 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b873a0 0x5576e9baaaa0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b869e0 0x5576e9b9f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/573007499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b84570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 -- 172.21.15.123:0/573007499 >> 172.21.15.123:0/573007499 conn(0x5576e9b80c00 msgr2=0x5576e9b83e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 -- 172.21.15.123:0/573007499 shutdown_connections 2024-03-24T00:40:09.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.529+0000 7fc0a7a78740 1 -- 172.21.15.123:0/573007499 wait complete. 2024-03-24T00:40:09.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.532+0000 7fc0a7a78740 1 -- start start 2024-03-24T00:40:09.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.534+0000 7fc0a7a78740 1 Processor -- start 2024-03-24T00:40:09.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.534+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b9ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a5710 con 0x5576e9b873a0 2024-03-24T00:40:09.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a6a30 con 0x5576e9b869e0 2024-03-24T00:40:09.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a5d70 con 0x5576e9b85810 2024-03-24T00:40:09.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc0a7a78740 10 client.0 _ll_register_callbacks cb 0x7fc090009350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:09.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.535+0000 7fc09de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b9ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.536+0000 7fc09e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.536+0000 7fc09e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36024/0 (socket says 172.21.15.123:36024) 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.536+0000 7fc09e800640 1 -- 172.21.15.123:0/1277430928 learned_addr learned my addr 172.21.15.123:0/1277430928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.536+0000 7fc09e800640 1 -- 172.21.15.123:0/1277430928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 msgr2=0x5576e9b9ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09d400640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09e800640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b9ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09e800640 1 -- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 msgr2=0x5576e9b9b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09e800640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b9b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09e800640 1 -- 172.21.15.123:0/1277430928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576e9b9e9c0 con 0x5576e9b873a0 2024-03-24T00:40:09.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.537+0000 7fc09d400640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b9b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:09.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.541+0000 7fc09e800640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc090012f10 tx=0x7fc0900138b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff3aa916cba322c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.545+0000 7fc07fe00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:09.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.552+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090034060 con 0x5576e9b873a0 2024-03-24T00:40:09.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.552+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc09001c640 con 0x5576e9b873a0 2024-03-24T00:40:09.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.552+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc090020030 con 0x5576e9b873a0 2024-03-24T00:40:09.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.553+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc090017040 con 0x5576e9b873a0 2024-03-24T00:40:09.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.553+0000 7fc0a7a78740 10 client.15567 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:09.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.553+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5576e9c10680 con 0x5576e9b873a0 2024-03-24T00:40:09.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.556+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc090020030 con 0x5576e9b873a0 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc090027430 con 0x5576e9b873a0 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 10 client.15567 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 10 client.15567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 20 client.15567 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9c10e20 0x5576e9c13240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 4 client.15567 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:40:09.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.557+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5576e9af1b50 con 0x5576e9c10e20 2024-03-24T00:40:09.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.559+0000 7fc09de00640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9c10e20 0x5576e9c13240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.562+0000 7fc09de00640 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9c10e20 0x5576e9c13240 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=145e10fb05e1b019 server_cookie=5990e9988b6df20d in_seq=0 out_seq=0 2024-03-24T00:40:09.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.562+0000 7fc07fe00640 10 client.15567 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:09.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.563+0000 7fc07fe00640 1 -- 172.21.15.123:0/1277430928 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5576e9af1b50 con 0x5576e9c10e20 2024-03-24T00:40:09.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.563+0000 7fc07fe00640 10 client.15567 handle_command_reply: tid=0 2024-03-24T00:40:09.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.563+0000 7fc0a7a78740 1 client.15567 shutdown 2024-03-24T00:40:09.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.566+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 msgr2=0x5576e9b9e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.567+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc090012f10 tx=0x7fc0900138b0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 shutdown_connections 2024-03-24T00:40:09.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5576e9b873a0 0x5576e9b9e3d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5576e9b869e0 0x5576e9b9b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5576e9c10e20 0x5576e9c13240 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 --2- 172.21.15.123:0/1277430928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e9b85810 0x5576e9b9ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.569+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 >> 172.21.15.123:0/1277430928 conn(0x5576e9b80c00 msgr2=0x5576e9b846f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:09 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1501721130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:09.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.576+0000 7fc07fe00640 0 client.15567 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:09.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.577+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 shutdown_connections 2024-03-24T00:40:09.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.577+0000 7fc0a7a78740 1 -- 172.21.15.123:0/1277430928 wait complete. 2024-03-24T00:40:09.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.579+0000 7fc0a7a78740 20 client.15567 trim_cache size 0 max 16384 2024-03-24T00:40:09.581 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 msgr2=0x7fc0840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 0x7fc0840a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc098037040 tx=0x7fc098000f40 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 msgr2=0x7fc0a019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00e9f0 tx=0x7fc08c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 shutdown_connections 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0a0103260 0x7fc0a01a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a01028a0 0x7fc0a019a200 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0840a2800 0x7fc0840a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 --2- 172.21.15.123:0/2277538388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0a01016b0 0x7fc0a0199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.582+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 >> 172.21.15.123:0/2277538388 conn(0x7fc0a00fcc40 msgr2=0x7fc0a00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.583+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 shutdown_connections 2024-03-24T00:40:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:09.583+0000 7fc0a4e00640 1 -- 172.21.15.123:0/2277538388 wait complete. 2024-03-24T00:40:09.604 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:09 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1501721130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:09 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1501721130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:10 smithi204 ceph-mon[36345]: pgmap v453: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 251 MiB/s wr, 103 op/s 2024-03-24T00:40:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:10 smithi123 ceph-mon[31487]: pgmap v453: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 251 MiB/s wr, 103 op/s 2024-03-24T00:40:10.606 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:40:10.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:10.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:40:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:10 smithi177 ceph-mon[36034]: pgmap v453: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 251 MiB/s wr, 103 op/s 2024-03-24T00:40:10.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.888+0000 7f87b5a00640 1 -- 172.21.15.123:0/784212072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 msgr2=0x7f87b010f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.888+0000 7f87b5a00640 1 --2- 172.21.15.123:0/784212072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b010f9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f87a8013490 tx=0x7f87a8034350 comp rx=0 tx=0).stop 2024-03-24T00:40:10.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.892+0000 7f87b5a00640 1 -- 172.21.15.123:0/784212072 shutdown_connections 2024-03-24T00:40:10.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.892+0000 7f87b5a00640 1 --2- 172.21.15.123:0/784212072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b010ff20 0x7f87b0112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.892+0000 7f87b5a00640 1 --2- 172.21.15.123:0/784212072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b010f9e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.892+0000 7f87b5a00640 1 --2- 172.21.15.123:0/784212072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b01086a0 0x7f87b0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.892+0000 7f87b5a00640 1 -- 172.21.15.123:0/784212072 >> 172.21.15.123:0/784212072 conn(0x7f87b006e8a0 msgr2=0x7f87b0070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.899+0000 7f87b5a00640 1 -- 172.21.15.123:0/784212072 shutdown_connections 2024-03-24T00:40:10.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.900+0000 7f87b5a00640 1 -- 172.21.15.123:0/784212072 wait complete. 2024-03-24T00:40:10.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.901+0000 7f87b5a00640 1 Processor -- start 2024-03-24T00:40:10.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.907+0000 7f87b5a00640 1 -- start start 2024-03-24T00:40:10.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.908+0000 7f87b5a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b01086a0 0x7f87b01af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.913+0000 7f87b5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b01af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.913+0000 7f87b5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.913+0000 7f87b5a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0113a50 con 0x7f87b01086a0 2024-03-24T00:40:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.913+0000 7f87b5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01138d0 con 0x7f87b0109000 2024-03-24T00:40:10.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.913+0000 7f87b5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0113bd0 con 0x7f87b010ff20 2024-03-24T00:40:10.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:41958/0 (socket says 172.21.15.123:41958) 2024-03-24T00:40:10.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 -- 172.21.15.123:0/3386988180 learned_addr learned my addr 172.21.15.123:0/3386988180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 -- 172.21.15.123:0/3386988180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 msgr2=0x7f87b01af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87ae000640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b01af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b01af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 -- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b01086a0 msgr2=0x7f87b01af230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b01086a0 0x7f87b01af230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87af400640 1 -- 172.21.15.123:0/3386988180 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b01b6e00 con 0x7f87b010ff20 2024-03-24T00:40:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.914+0000 7f87ae000640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b01af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:10.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.915+0000 7f87af400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b3f0 tx=0x7f87a000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.916+0000 7f87a7400640 1 -- 172.21.15.123:0/3386988180 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0015020 con 0x7f87b010ff20 2024-03-24T00:40:10.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.916+0000 7f87a7400640 1 -- 172.21.15.123:0/3386988180 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a0011d10 con 0x7f87b010ff20 2024-03-24T00:40:10.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.916+0000 7f87a7400640 1 -- 172.21.15.123:0/3386988180 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a000f2e0 con 0x7f87b010ff20 2024-03-24T00:40:10.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.917+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b01b70f0 con 0x7f87b010ff20 2024-03-24T00:40:10.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.917+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b01b7560 con 0x7f87b010ff20 2024-03-24T00:40:10.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.928+0000 7f87a7400640 1 -- 172.21.15.123:0/3386988180 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f87a0018020 con 0x7f87b010ff20 2024-03-24T00:40:10.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.930+0000 7f87b8779740 1 Processor -- start 2024-03-24T00:40:10.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.930+0000 7f87b8779740 1 -- start start 2024-03-24T00:40:10.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1134c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc113a90 0x5595fc118590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc078eb0 con 0x5595fc1130a0 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc021a60 con 0x5595fc118c00 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87b8779740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc074f50 con 0x5595fc113a90 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.931+0000 7f87a7400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 0x7f87940a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56508/0 (socket says 172.21.15.123:56508) 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 -- 172.21.15.123:0/267092207 learned_addr learned my addr 172.21.15.123:0/267092207 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87aea00640 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc113a90 0x5595fc118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87af400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 0x7f87940a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 -- 172.21.15.123:0/267092207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc113a90 msgr2=0x5595fc118590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc113a90 0x5595fc118590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 -- 172.21.15.123:0/267092207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 msgr2=0x5595fc1134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.932+0000 7f87ae000640 1 -- 172.21.15.123:0/267092207 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595fbe557e0 con 0x5595fc118c00 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.933+0000 7f87a7400640 1 -- 172.21.15.123:0/3386988180 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f87a00ce170 con 0x7f87b010ff20 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.933+0000 7f87ae000640 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f87a8014090 tx=0x7f87a8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26dbaf44a6316275 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.933+0000 7f879be00640 1 -- 172.21.15.123:0/267092207 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a803c940 con 0x5595fc118c00 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.933+0000 7f879be00640 1 -- 172.21.15.123:0/267092207 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a8003e50 con 0x5595fc118c00 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.934+0000 7f87b8779740 1 -- 172.21.15.123:0/267092207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 msgr2=0x5595fc11b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.934+0000 7f87b8779740 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f87a8014090 tx=0x7f87a8014000 comp rx=0 tx=0).stop 2024-03-24T00:40:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.934+0000 7f87af400640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 0x7f87940a4d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f87a00d1fd0 tx=0x7f87a00d1960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 -- 172.21.15.123:0/267092207 shutdown_connections 2024-03-24T00:40:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc11b090 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc113a90 0x5595fc118590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 --2- 172.21.15.123:0/267092207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 -- 172.21.15.123:0/267092207 >> 172.21.15.123:0/267092207 conn(0x5595fc10e610 msgr2=0x5595fc10ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 -- 172.21.15.123:0/267092207 shutdown_connections 2024-03-24T00:40:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.936+0000 7f87b8779740 1 -- 172.21.15.123:0/267092207 wait complete. 2024-03-24T00:40:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.938+0000 7f87b8779740 1 -- start start 2024-03-24T00:40:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 Processor -- start 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc11d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc124810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc11bba0 con 0x5595fc113a90 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc034eb0 con 0x5595fc118c00 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595fc11bd20 con 0x5595fc1130a0 2024-03-24T00:40:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.939+0000 7f87b8779740 10 client.0 _ll_register_callbacks cb 0x5595fc029150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.940+0000 7f87ae000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.940+0000 7f87ae000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc124810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.941+0000 7f87ae000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:41980/0 (socket says 172.21.15.123:41980) 2024-03-24T00:40:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.941+0000 7f87ae000640 1 -- 172.21.15.123:0/3371472635 learned_addr learned my addr 172.21.15.123:0/3371472635 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.941+0000 7f87aea00640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc11d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87ae000640 1 -- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 msgr2=0x5595fc124810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87ae000640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc124810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87ae000640 1 -- 172.21.15.123:0/3371472635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 msgr2=0x5595fc11d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87ae000640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc11d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87ae000640 1 -- 172.21.15.123:0/3371472635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595fbeb5f70 con 0x5595fc1130a0 2024-03-24T00:40:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.942+0000 7f87aea00640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc11d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:10.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.944+0000 7f87ae000640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87a8002230 tx=0x7f87a8000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6006b65f314854b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.944+0000 7f8798c00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:10.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.950+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8003b80 con 0x5595fc1130a0 2024-03-24T00:40:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.950+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f87a803c740 con 0x5595fc1130a0 2024-03-24T00:40:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.953+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87a8003e50 con 0x5595fc1130a0 2024-03-24T00:40:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.953+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87a803c940 con 0x5595fc1130a0 2024-03-24T00:40:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.953+0000 7f87b8779740 10 client.25675 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.953+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5595fbd4d170 con 0x5595fc1130a0 2024-03-24T00:40:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.954+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a8003e50 con 0x5595fc1130a0 2024-03-24T00:40:10.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.955+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f87a80456b0 con 0x5595fc1130a0 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 10 client.25675 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 10 client.25675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 20 client.25675 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc128d00 0x5595fc12b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 4 client.25675 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.956+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595fc0863f0 con 0x5595fc128d00 2024-03-24T00:40:10.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.957+0000 7f87aea00640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc128d00 0x5595fc12b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.964+0000 7f87aea00640 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc128d00 0x5595fc12b0f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41de4bb9cb031a48 server_cookie=b9e88745ba72b81f in_seq=0 out_seq=0 2024-03-24T00:40:10.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.964+0000 7f8798c00640 10 client.25675 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:10.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.965+0000 7f8798c00640 1 -- 172.21.15.123:0/3371472635 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5595fc0863f0 con 0x5595fc128d00 2024-03-24T00:40:10.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.965+0000 7f8798c00640 10 client.25675 handle_command_reply: tid=0 2024-03-24T00:40:10.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.965+0000 7f87b8779740 1 client.25675 shutdown 2024-03-24T00:40:10.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.966+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 msgr2=0x5595fc11d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.966+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87a8002230 tx=0x7f87a8000f40 comp rx=0 tx=0).stop 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.969+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 shutdown_connections 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc124810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc11d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc128d00 0x5595fc12b0f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f87b8779740 1 --2- 172.21.15.123:0/3371472635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc11d290 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 >> 172.21.15.123:0/3371472635 conn(0x5595fc10e610 msgr2=0x5595fc10d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.970+0000 7f8798c00640 0 client.25675 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:10.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.971+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 shutdown_connections 2024-03-24T00:40:10.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.972+0000 7f87b8779740 1 -- 172.21.15.123:0/3371472635 wait complete. 2024-03-24T00:40:10.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:10.972+0000 7f87b8779740 20 client.25675 trim_cache size 0 max 16384 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 Processor -- start 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 -- start start 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc114530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc1247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01be3f0 con 0x5595fc113a90 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01c09b0 con 0x5595fc118c00 2024-03-24T00:40:11.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.011+0000 7f87b8779740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01be9f0 con 0x5595fc1130a0 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87aea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc1247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56528/0 (socket says 172.21.15.123:56528) 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 -- 172.21.15.123:0/3638142829 learned_addr learned my addr 172.21.15.123:0/3638142829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 -- 172.21.15.123:0/3638142829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 msgr2=0x5595fc114530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc114530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.012+0000 7f87ae000640 1 -- 172.21.15.123:0/3638142829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 msgr2=0x5595fc1247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.013+0000 7f87ae000640 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc1247a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.013+0000 7f87ae000640 1 -- 172.21.15.123:0/3638142829 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595fc10df60 con 0x5595fc118c00 2024-03-24T00:40:11.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.013+0000 7f87aea00640 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc1247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:11.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.013+0000 7f87ae000640 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f87a8045fd0 tx=0x7f87a803ce40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87809ad4fa6a171b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.015+0000 7f879b400640 1 -- 172.21.15.123:0/3638142829 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8049070 con 0x5595fc118c00 2024-03-24T00:40:11.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.015+0000 7f879b400640 1 -- 172.21.15.123:0/3638142829 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a803c740 con 0x5595fc118c00 2024-03-24T00:40:11.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.015+0000 7f879b400640 1 -- 172.21.15.123:0/3638142829 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8045c70 con 0x5595fc118c00 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 -- 172.21.15.123:0/3638142829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 msgr2=0x5595fc114a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f87a8045fd0 tx=0x7f87a803ce40 comp rx=0 tx=0).stop 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 -- 172.21.15.123:0/3638142829 shutdown_connections 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc118c00 0x5595fc114a70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc113a90 0x5595fc1247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 --2- 172.21.15.123:0/3638142829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc1130a0 0x5595fc114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 -- 172.21.15.123:0/3638142829 >> 172.21.15.123:0/3638142829 conn(0x5595fc10e610 msgr2=0x5595fc10d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 -- 172.21.15.123:0/3638142829 shutdown_connections 2024-03-24T00:40:11.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.016+0000 7f87b8779740 1 -- 172.21.15.123:0/3638142829 wait complete. 2024-03-24T00:40:11.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.017+0000 7f87b8779740 1 -- start start 2024-03-24T00:40:11.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.017+0000 7f87b8779740 1 Processor -- start 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 0x5595fc122d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc118c00 0x5595fc123280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01bf030 con 0x5595fc1130a0 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01c0350 con 0x5595fc113a90 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01bf690 con 0x5595fc118c00 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87b8779740 10 client.0 _ll_register_callbacks cb 0x7f87a8013120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87ae000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87ae000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36066/0 (socket says 172.21.15.123:36066) 2024-03-24T00:40:11.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.018+0000 7f87ae000640 1 -- 172.21.15.123:0/925202122 learned_addr learned my addr 172.21.15.123:0/925202122 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87aea00640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 0x5595fc122d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc118c00 0x5595fc123280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 -- 172.21.15.123:0/925202122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc118c00 msgr2=0x5595fc123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc118c00 0x5595fc123280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 -- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 msgr2=0x5595fc122d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 0x5595fc122d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 -- 172.21.15.123:0/925202122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595fc109c20 con 0x5595fc1130a0 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87aea00640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 0x5595fc122d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f87ae000640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f87a8004010 tx=0x7f87a8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435ea37567016a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.019+0000 7f8798c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:11.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8049050 con 0x5595fc1130a0 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f87a80043f0 con 0x5595fc1130a0 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87a803c740 con 0x5595fc1130a0 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87a80110b0 con 0x5595fc1130a0 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f87b8779740 10 client.15573 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.020+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5595fc1851b0 con 0x5595fc1130a0 2024-03-24T00:40:11.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a803c740 con 0x5595fc1130a0 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f87a805d670 con 0x5595fc1130a0 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 10 client.15573 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 20 client.15573 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc185820 0x5595fc187c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 4 client.15573 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.021+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5595fc080740 con 0x5595fc185820 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.022+0000 7f87aea00640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc185820 0x5595fc187c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.022+0000 7f8798c00640 10 client.15573 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.022+0000 7f87aea00640 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc185820 0x5595fc187c40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ce3dc7ff3c9687 server_cookie=135f90af00a9b51c in_seq=0 out_seq=0 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.023+0000 7f8798c00640 1 -- 172.21.15.123:0/925202122 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5595fc080740 con 0x5595fc185820 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.023+0000 7f8798c00640 10 client.15573 handle_command_reply: tid=0 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.023+0000 7f87b8779740 1 client.15573 shutdown 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 msgr2=0x5595fc1246e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f87a8004010 tx=0x7f87a8000f40 comp rx=0 tx=0).stop 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 shutdown_connections 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5595fc185820 0x5595fc187c40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595fc118c00 0x5595fc123280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595fc113a90 0x5595fc122d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 --2- 172.21.15.123:0/925202122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595fc1130a0 0x5595fc1246e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 >> 172.21.15.123:0/925202122 conn(0x5595fc10e610 msgr2=0x5595fc111870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f8798c00640 0 client.15573 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.024+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 shutdown_connections 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.025+0000 7f87b8779740 1 -- 172.21.15.123:0/925202122 wait complete. 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.025+0000 7f87b8779740 20 client.15573 trim_cache size 0 max 16384 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "898b7d6f-0910-4231-9705-740681b4c02e", 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:40:11.025 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:11.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 msgr2=0x7f87940a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 0x7f87940a4d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f87a00d1fd0 tx=0x7f87a00d1960 comp rx=0 tx=0).stop 2024-03-24T00:40:11.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 msgr2=0x7f87b01b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b3f0 tx=0x7f87a000b8b0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 shutdown_connections 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b010ff20 0x7f87b01b6790 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87b0109000 0x7f87b01af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f87940a28d0 0x7f87940a4d90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 --2- 172.21.15.123:0/3386988180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b01086a0 0x7f87b01af230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 >> 172.21.15.123:0/3386988180 conn(0x7f87b006e8a0 msgr2=0x7f87b010b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 shutdown_connections 2024-03-24T00:40:11.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.026+0000 7f87b5a00640 1 -- 172.21.15.123:0/3386988180 wait complete. 2024-03-24T00:40:11.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:40:11.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:40:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 -- 172.21.15.123:0/4181500150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde98103260 msgr2=0x7fde98109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 --2- 172.21.15.123:0/4181500150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde98103260 0x7fde98109af0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fde90009ca0 tx=0x7fde9002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 -- 172.21.15.123:0/4181500150 shutdown_connections 2024-03-24T00:40:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 --2- 172.21.15.123:0/4181500150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde98103260 0x7fde98109af0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 --2- 172.21.15.123:0/4181500150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981028a0 0x7fde98102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 --2- 172.21.15.123:0/4181500150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde981016b0 0x7fde98101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 -- 172.21.15.123:0/4181500150 >> 172.21.15.123:0/4181500150 conn(0x7fde980fcc40 msgr2=0x7fde980ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 -- 172.21.15.123:0/4181500150 shutdown_connections 2024-03-24T00:40:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.315+0000 7fde9de00640 1 -- 172.21.15.123:0/4181500150 wait complete. 2024-03-24T00:40:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.316+0000 7fde9de00640 1 Processor -- start 2024-03-24T00:40:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.316+0000 7fde9de00640 1 -- start start 2024-03-24T00:40:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.316+0000 7fde9de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde9de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde981028a0 0x7fde98073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde9de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 0x7fde98073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde9de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810d330 con 0x7fde98103260 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde9de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810d1b0 con 0x7fde981028a0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde9de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9810d4b0 con 0x7fde981016b0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42014/0 (socket says 172.21.15.123:42014) 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde981028a0 0x7fde98073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 -- 172.21.15.123:0/117010362 learned_addr learned my addr 172.21.15.123:0/117010362 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde97400640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 0x7fde98073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 -- 172.21.15.123:0/117010362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde981028a0 msgr2=0x7fde98073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde981028a0 0x7fde98073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 -- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 msgr2=0x7fde98073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 0x7fde98073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde96a00640 1 -- 172.21.15.123:0/117010362 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde98074860 con 0x7fde981016b0 2024-03-24T00:40:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.317+0000 7fde97400640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 0x7fde98073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde96a00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d950 tx=0x7fde8800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde9ca00640 1 -- 172.21.15.123:0/117010362 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88017020 con 0x7fde981016b0 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde9ca00640 1 -- 172.21.15.123:0/117010362 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde88009c30 con 0x7fde981016b0 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde9ca00640 1 -- 172.21.15.123:0/117010362 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8800f720 con 0x7fde981016b0 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde981a1a90 con 0x7fde981016b0 2024-03-24T00:40:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.318+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde981a1f90 con 0x7fde981016b0 2024-03-24T00:40:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.321+0000 7fde9ca00640 1 -- 172.21.15.123:0/117010362 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fde8801a020 con 0x7fde981016b0 2024-03-24T00:40:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.321+0000 7fdea0a82740 1 Processor -- start 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 -- start start 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311110ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311115b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231106d250 con 0x562311110690 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562311015a30 con 0x5623111161f0 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fdea0a82740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623110692f0 con 0x562311111080 2024-03-24T00:40:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fde96000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fde96a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fde96000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56548/0 (socket says 172.21.15.123:56548) 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fde96000640 1 -- 172.21.15.123:0/3493777896 learned_addr learned my addr 172.21.15.123:0/3493777896 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.322+0000 7fde97400640 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311110ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 -- 172.21.15.123:0/3493777896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 msgr2=0x562311115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 -- 172.21.15.123:0/3493777896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 msgr2=0x562311110ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311110ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 -- 172.21.15.123:0/3493777896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562311000690 con 0x5623111161f0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde97400640 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde96000640 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fde84009c20 tx=0x7fde84011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5573faf50fc6f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde9ca00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 0x7fde7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde83e00640 1 -- 172.21.15.123:0/3493777896 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84004230 con 0x5623111161f0 2024-03-24T00:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde83e00640 1 -- 172.21.15.123:0/3493777896 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde840043d0 con 0x5623111161f0 2024-03-24T00:40:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde83e00640 1 -- 172.21.15.123:0/3493777896 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84004db0 con 0x5623111161f0 2024-03-24T00:40:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde9ca00640 1 -- 172.21.15.123:0/117010362 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fde880ce9c0 con 0x7fde981016b0 2024-03-24T00:40:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.323+0000 7fde97400640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 0x7fde7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.324+0000 7fdea0a82740 1 -- 172.21.15.123:0/3493777896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 msgr2=0x562311118680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.324+0000 7fde97400640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 0x7fde7c0a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fde900324c0 tx=0x7fde90004a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.324+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fde84009c20 tx=0x7fde84011810 comp rx=0 tx=0).stop 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 -- 172.21.15.123:0/3493777896 shutdown_connections 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311118680 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3493777896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 -- 172.21.15.123:0/3493777896 >> 172.21.15.123:0/3493777896 conn(0x5623110d0200 msgr2=0x56231110c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 -- 172.21.15.123:0/3493777896 shutdown_connections 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 -- 172.21.15.123:0/3493777896 wait complete. 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 -- start start 2024-03-24T00:40:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.325+0000 7fdea0a82740 1 Processor -- start 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x56231111a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x56231111ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311121be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623111191a0 con 0x562311110690 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562311028e80 con 0x5623111161f0 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562311119320 con 0x562311111080 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fde96000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x56231111ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fde96000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x56231111ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42030/0 (socket says 172.21.15.123:42030) 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fde96000640 1 -- 172.21.15.123:0/1064567849 learned_addr learned my addr 172.21.15.123:0/1064567849 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fde97400640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311121be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.326+0000 7fdea0a82740 10 client.0 _ll_register_callbacks cb 0x56231103b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde96a00640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x56231111a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 -- 172.21.15.123:0/1064567849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 msgr2=0x56231111ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x56231111ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 -- 172.21.15.123:0/1064567849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 msgr2=0x56231111a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x56231111a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 -- 172.21.15.123:0/1064567849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562310e4b7d0 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde97400640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311121be0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde9002c630 tx=0x7fde9002c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37863af03cf6d746 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde80c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9004f070 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fde900040c0 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.327+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde840043d0 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde90039040 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 10 client.25694 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562310d813f0 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde840043d0 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fde9004a070 con 0x5623111161f0 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 10 client.25694 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 10 client.25694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 20 client.25694 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111263e0 0x562311128870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 4 client.25694 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.328+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56231107a790 con 0x5623111263e0 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.329+0000 7fde96a00640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111263e0 0x562311128870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.329+0000 7fde96a00640 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111263e0 0x562311128870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92d130837c4db2a7 server_cookie=6d4f880dcb9e366b in_seq=0 out_seq=0 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.329+0000 7fde80c00640 10 client.25694 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fde80c00640 1 -- 172.21.15.123:0/1064567849 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fde880804b0 con 0x5623111263e0 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fde80c00640 10 client.25694 handle_command_reply: tid=0 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 client.25694 shutdown 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 msgr2=0x562311121be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311121be0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde9002c630 tx=0x7fde9002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 shutdown_connections 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x562311121be0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111263e0 0x562311128870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x56231111ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1064567849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x56231111a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fde80c00640 0 client.25694 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 >> 172.21.15.123:0/1064567849 conn(0x5623110d0200 msgr2=0x562311102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 shutdown_connections 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 1 -- 172.21.15.123:0/1064567849 wait complete. 2024-03-24T00:40:11.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.330+0000 7fdea0a82740 20 client.25694 trim_cache size 0 max 16384 2024-03-24T00:40:11.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 Processor -- start 2024-03-24T00:40:11.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 -- start start 2024-03-24T00:40:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311121ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x56231112d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a48c0 con 0x562311110690 2024-03-24T00:40:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a6e80 con 0x5623111161f0 2024-03-24T00:40:11.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.350+0000 7fdea0a82740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a4ec0 con 0x562311111080 2024-03-24T00:40:11.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36112/0 (socket says 172.21.15.123:36112) 2024-03-24T00:40:11.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 -- 172.21.15.123:0/3836507809 learned_addr learned my addr 172.21.15.123:0/3836507809 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96000640 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311121ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 -- 172.21.15.123:0/3836507809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 msgr2=0x562311121ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311121ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 -- 172.21.15.123:0/3836507809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 msgr2=0x56231112d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x56231112d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.351+0000 7fde96a00640 1 -- 172.21.15.123:0/3836507809 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562310d410f0 con 0x562311110690 2024-03-24T00:40:11.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.352+0000 7fde96a00640 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fde88097ce0 tx=0x7fde880d4530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e107ed4a1ac0910 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.355+0000 7fde83400640 1 -- 172.21.15.123:0/3836507809 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8809b310 con 0x562311110690 2024-03-24T00:40:11.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.355+0000 7fde83400640 1 -- 172.21.15.123:0/3836507809 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56231107a790 con 0x562311110690 2024-03-24T00:40:11.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.355+0000 7fde83400640 1 -- 172.21.15.123:0/3836507809 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8809f130 con 0x562311110690 2024-03-24T00:40:11.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 -- 172.21.15.123:0/3836507809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 msgr2=0x562311112230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fde88097ce0 tx=0x7fde880d4530 comp rx=0 tx=0).stop 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 -- 172.21.15.123:0/3836507809 shutdown_connections 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623111161f0 0x56231112d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311111080 0x562311121ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 --2- 172.21.15.123:0/3836507809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562311110690 0x562311112230 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 -- 172.21.15.123:0/3836507809 >> 172.21.15.123:0/3836507809 conn(0x5623110d0200 msgr2=0x562311111b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 -- 172.21.15.123:0/3836507809 shutdown_connections 2024-03-24T00:40:11.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:11 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:40:11.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:11 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:11.357 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:11 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:40:11.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.356+0000 7fdea0a82740 1 -- 172.21.15.123:0/3836507809 wait complete. 2024-03-24T00:40:11.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.357+0000 7fdea0a82740 1 -- start start 2024-03-24T00:40:11.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.357+0000 7fdea0a82740 1 Processor -- start 2024-03-24T00:40:11.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.358+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.358+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 0x562311120000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.358+0000 7fdea0a82740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde96a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde96a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42058/0 (socket says 172.21.15.123:42058) 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36124/0 (socket says 172.21.15.123:36124) 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 -- 172.21.15.123:0/1545405697 learned_addr learned my addr 172.21.15.123:0/1545405697 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a5500 con 0x5623111161f0 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a6820 con 0x562311111080 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981a5b60 con 0x562311110690 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde96000640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 0x562311120000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fdea0a82740 10 client.0 _ll_register_callbacks cb 0x7fde90037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 -- 172.21.15.123:0/1545405697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 msgr2=0x5623111219f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 -- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 msgr2=0x562311120000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.359+0000 7fde97400640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 0x562311120000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde97400640 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562311120cb0 con 0x5623111161f0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde96a00640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde96000640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 0x562311120000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde97400640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde90004af0 tx=0x7fde90009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e109bca50ba27a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde80c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde90064060 con 0x5623111161f0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.360+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fde9003f440 con 0x5623111161f0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.361+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56231107a790 con 0x5623111161f0 2024-03-24T00:40:11.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.361+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde9004f040 con 0x5623111161f0 2024-03-24T00:40:11.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.361+0000 7fdea0a82740 10 client.15588 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:11.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.361+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562311182d20 con 0x5623111161f0 2024-03-24T00:40:11.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.361+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56231107a790 con 0x5623111161f0 2024-03-24T00:40:11.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fde9004a070 con 0x5623111161f0 2024-03-24T00:40:11.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 10 client.15588 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 20 client.15588 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111836b0 0x562311185ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 4 client.15588 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562311074ae0 con 0x5623111836b0 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.362+0000 7fde96a00640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111836b0 0x562311185ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fde96a00640 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111836b0 0x562311185ad0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27b1dae77c993cc server_cookie=1e396d591f8ec907 in_seq=0 out_seq=0 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fde80c00640 10 client.15588 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fde80c00640 1 -- 172.21.15.123:0/1545405697 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562311074ae0 con 0x5623111836b0 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fde80c00640 10 client.15588 handle_command_reply: tid=0 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 client.15588 shutdown 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 msgr2=0x562311120540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde90004af0 tx=0x7fde90009b60 comp rx=0 tx=0).stop 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 shutdown_connections 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623111161f0 0x562311120540 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562311111080 0x562311120000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623111836b0 0x562311185ad0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 --2- 172.21.15.123:0/1545405697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562311110690 0x5623111219f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 >> 172.21.15.123:0/1545405697 conn(0x5623110d0200 msgr2=0x56231110fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.363+0000 7fde80c00640 0 client.15588 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 shutdown_connections 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fdea0a82740 1 -- 172.21.15.123:0/1545405697 wait complete. 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fdea0a82740 20 client.15588 trim_cache size 0 max 16384 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:40:11.365 INFO:teuthology.orchestra.run.smithi123.stdout: "898b7d6f-0910-4231-9705-740681b4c02e": { 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "898b7d6f-0910-4231-9705-740681b4c02e", 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 msgr2=0x7fde7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 0x7fde7c0a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fde900324c0 tx=0x7fde90004a60 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 msgr2=0x7fde98075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d950 tx=0x7fde8800de10 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 shutdown_connections 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fde7c0a2880 0x7fde7c0a4d40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde98103260 0x7fde98073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde981028a0 0x7fde98073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 --2- 172.21.15.123:0/117010362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde981016b0 0x7fde98075450 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d950 tx=0x7fde8800de10 comp rx=0 tx=0).stop 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.364+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 >> 172.21.15.123:0/117010362 conn(0x7fde980fcc40 msgr2=0x7fde98105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.365+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 shutdown_connections 2024-03-24T00:40:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:11.365+0000 7fde9de00640 1 -- 172.21.15.123:0/117010362 wait complete. 2024-03-24T00:40:11.387 INFO:tasks.cephfs.filesystem:scrub status for tag:898b7d6f-0910-4231-9705-740681b4c02e - {'path': '/', 'tag': '898b7d6f-0910-4231-9705-740681b4c02e', 'options': 'recursive,force'} 2024-03-24T00:40:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:11 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:40:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:11 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:11.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:11 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:40:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:11 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:40:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:11 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:11 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 -- 172.21.15.123:0/3681742386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 msgr2=0x7f73e0073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 --2- 172.21.15.123:0/3681742386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e0073b30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009930 tx=0x7f73d002bc20 comp rx=0 tx=0).stop 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 -- 172.21.15.123:0/3681742386 shutdown_connections 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 --2- 172.21.15.123:0/3681742386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73e010dae0 0x7f73e010fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 --2- 172.21.15.123:0/3681742386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 0x7f73e0074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 --2- 172.21.15.123:0/3681742386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e0073b30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.920+0000 7f73e4e00640 1 -- 172.21.15.123:0/3681742386 >> 172.21.15.123:0/3681742386 conn(0x7f73e00fca80 msgr2=0x7f73e00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.921 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.921+0000 7f73e4e00640 1 -- 172.21.15.123:0/3681742386 shutdown_connections 2024-03-24T00:40:11.921 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.921+0000 7f73e4e00640 1 -- 172.21.15.123:0/3681742386 wait complete. 2024-03-24T00:40:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.923+0000 7f73e4e00640 1 Processor -- start 2024-03-24T00:40:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.924+0000 7f73e4e00640 1 -- start start 2024-03-24T00:40:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.926+0000 7f73e4e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 0x7f73e019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.926+0000 7f73e4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73e4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73e010dae0 0x7f73e01a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73e4e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01137b0 con 0x7f73e010dae0 2024-03-24T00:40:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73e4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0113630 con 0x7f73e00756e0 2024-03-24T00:40:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73e4e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0113930 con 0x7f73e0074100 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56590/0 (socket says 172.21.15.123:56590) 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 -- 172.21.15.123:0/599166108 learned_addr learned my addr 172.21.15.123:0/599166108 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73df400640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 0x7f73e019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 -- 172.21.15.123:0/599166108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 msgr2=0x7f73e019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 0x7f73e019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 -- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73e010dae0 msgr2=0x7f73e01a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73e010dae0 0x7f73e01a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.927+0000 7f73dea00640 1 -- 172.21.15.123:0/599166108 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e01a5a40 con 0x7f73e00756e0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73dea00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00d950 tx=0x7f73cc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73d7e00640 1 -- 172.21.15.123:0/599166108 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc017070 con 0x7f73e00756e0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e01a5d30 con 0x7f73e00756e0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73d7e00640 1 -- 172.21.15.123:0/599166108 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73cc009c30 con 0x7f73e00756e0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73d7e00640 1 -- 172.21.15.123:0/599166108 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc00f6b0 con 0x7f73e00756e0 2024-03-24T00:40:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.928+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01a6270 con 0x7f73e00756e0 2024-03-24T00:40:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.933+0000 7f73d7e00640 1 -- 172.21.15.123:0/599166108 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f73cc0109d0 con 0x7f73e00756e0 2024-03-24T00:40:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.934+0000 7f73d7e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 0x7f73c40a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.934+0000 7f73d7e00640 1 -- 172.21.15.123:0/599166108 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73cc0ce990 con 0x7f73e00756e0 2024-03-24T00:40:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.935+0000 7f73e9187740 1 Processor -- start 2024-03-24T00:40:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.935+0000 7f73e9187740 1 -- start start 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.935+0000 7f73df400640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 0x7f73c40a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234ed3f0 0x55c3234ed810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c3234f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3234b03b0 con 0x55c3234edde0 2024-03-24T00:40:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c323452830 con 0x55c3234ed3f0 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73e9187740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3233fb5d0 con 0x55c3234f2f50 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73df400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234ed3f0 0x55c3234ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73dfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73dfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42088/0 (socket says 172.21.15.123:42088) 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73dfe00640 1 -- 172.21.15.123:0/4126746059 learned_addr learned my addr 172.21.15.123:0/4126746059 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73dea00640 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c3234f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.936+0000 7f73df400640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 0x7f73c40a4d70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f73d0036040 tx=0x7f73d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 -- 172.21.15.123:0/4126746059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234ed3f0 msgr2=0x55c3234ed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234ed3f0 0x55c3234ed810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 -- 172.21.15.123:0/4126746059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 msgr2=0x55c3234f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c3234f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 -- 172.21.15.123:0/4126746059 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3234132c0 con 0x55c3234f2f50 2024-03-24T00:40:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73dfe00640 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009ea0 tx=0x7f73d8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159270b9ac041b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73d4c00640 1 -- 172.21.15.123:0/4126746059 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8004d20 con 0x55c3234f2f50 2024-03-24T00:40:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73d4c00640 1 -- 172.21.15.123:0/4126746059 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d800ee60 con 0x55c3234f2f50 2024-03-24T00:40:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.937+0000 7f73d4c00640 1 -- 172.21.15.123:0/4126746059 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80218b0 con 0x55c3234f2f50 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.938+0000 7f73e9187740 1 -- 172.21.15.123:0/4126746059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 msgr2=0x55c3234f53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.938+0000 7f73e9187740 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009ea0 tx=0x7f73d8009f90 comp rx=0 tx=0).stop 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 -- 172.21.15.123:0/4126746059 shutdown_connections 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234f2f50 0x55c3234f53e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c3234f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 --2- 172.21.15.123:0/4126746059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234ed3f0 0x55c3234ed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 -- 172.21.15.123:0/4126746059 >> 172.21.15.123:0/4126746059 conn(0x55c3234e8960 msgr2=0x55c3234e8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 -- 172.21.15.123:0/4126746059 shutdown_connections 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 -- 172.21.15.123:0/4126746059 wait complete. 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 -- start start 2024-03-24T00:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.939+0000 7f73e9187740 1 Processor -- start 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c323508590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234f2f50 0x55c32350fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c32340ea20 con 0x55c3234edde0 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c32344e8d0 con 0x55c3234f2f50 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c323506f10 con 0x55c3234ed3f0 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73e9187740 10 client.0 _ll_register_callbacks cb 0x55c32344b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73dea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73dea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36158/0 (socket says 172.21.15.123:36158) 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73dea00640 1 -- 172.21.15.123:0/1343500203 learned_addr learned my addr 172.21.15.123:0/1343500203 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.940+0000 7f73dfe00640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234f2f50 0x55c32350fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 -- 172.21.15.123:0/1343500203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 msgr2=0x55c323508590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c323508590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 -- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234f2f50 msgr2=0x55c32350fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234f2f50 0x55c32350fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 -- 172.21.15.123:0/1343500203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c323167210 con 0x55c3234edde0 2024-03-24T00:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73dea00640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73cc010fd0 tx=0x7f73cc09bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a14ba86c5b45c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73c9600640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0b4060 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73cc0979b0 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.941+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d800ee60 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.942+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73cc020040 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.942+0000 7f73e9187740 10 client.15597 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.942+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c323233620 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.942+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d800ee60 con 0x55c3234edde0 2024-03-24T00:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f73cc0a3650 con 0x55c3234edde0 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 10 client.15597 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 20 client.15597 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c3235143f0 0x55c3235167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 4 client.15597 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c32345fd70 con 0x55c3235143f0 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.944+0000 7f73df400640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c3235143f0 0x55c3235167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.945+0000 7f73df400640 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c3235143f0 0x55c3235167e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb921fdcfe8f2111 server_cookie=36529a36f8cb73eb in_seq=0 out_seq=0 2024-03-24T00:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.945+0000 7f73c9600640 10 client.15597 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.945+0000 7f73c9600640 1 -- 172.21.15.123:0/1343500203 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c32345fd70 con 0x55c3235143f0 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.945+0000 7f73c9600640 10 client.15597 handle_command_reply: tid=0 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.945+0000 7f73e9187740 1 client.15597 shutdown 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 msgr2=0x55c323508ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73cc010fd0 tx=0x7f73cc09bb00 comp rx=0 tx=0).stop 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 shutdown_connections 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234f2f50 0x55c32350fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234edde0 0x55c323508ad0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c3235143f0 0x55c3235167e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 --2- 172.21.15.123:0/1343500203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c323508590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 >> 172.21.15.123:0/1343500203 conn(0x55c3234e8960 msgr2=0x55c3234e7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73c9600640 0 client.15597 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 shutdown_connections 2024-03-24T00:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 1 -- 172.21.15.123:0/1343500203 wait complete. 2024-03-24T00:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.946+0000 7f73e9187740 20 client.15597 trim_cache size 0 max 16384 2024-03-24T00:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.969+0000 7f73e9187740 1 Processor -- start 2024-03-24T00:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.969+0000 7f73e9187740 1 -- start start 2024-03-24T00:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.969+0000 7f73e9187740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.969+0000 7f73e9187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73e9187740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32351b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73e9187740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0071bf0 con 0x55c3234f2f50 2024-03-24T00:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73df400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73e9187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0105bb0 con 0x55c3234edde0 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73e9187740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00721f0 con 0x55c3234ed3f0 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73df400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42104/0 (socket says 172.21.15.123:42104) 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73df400640 1 -- 172.21.15.123:0/2530665688 learned_addr learned my addr 172.21.15.123:0/2530665688 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.970+0000 7f73dea00640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 -- 172.21.15.123:0/2530665688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 msgr2=0x55c32350fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 -- 172.21.15.123:0/2530665688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 msgr2=0x55c32351b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73dfe00640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32351b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32351b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 -- 172.21.15.123:0/2530665688 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c323509430 con 0x55c3234ed3f0 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73dfe00640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32351b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.971+0000 7f73df400640 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f73d002c230 tx=0x7f73d0046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57f6db8cbfb681f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.972+0000 7f73cbe00640 1 -- 172.21.15.123:0/2530665688 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d005a070 con 0x55c3234ed3f0 2024-03-24T00:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.972+0000 7f73cbe00640 1 -- 172.21.15.123:0/2530665688 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d002cc00 con 0x55c3234ed3f0 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.972+0000 7f73cbe00640 1 -- 172.21.15.123:0/2530665688 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0004c00 con 0x55c3234ed3f0 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 -- 172.21.15.123:0/2530665688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 msgr2=0x55c32351b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f73d002c230 tx=0x7f73d0046000 comp rx=0 tx=0).stop 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 -- 172.21.15.123:0/2530665688 shutdown_connections 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32351b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 --2- 172.21.15.123:0/2530665688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32351b0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 -- 172.21.15.123:0/2530665688 >> 172.21.15.123:0/2530665688 conn(0x55c3234e8960 msgr2=0x55c3234ebbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 -- 172.21.15.123:0/2530665688 shutdown_connections 2024-03-24T00:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.973+0000 7f73e9187740 1 -- 172.21.15.123:0/2530665688 wait complete. 2024-03-24T00:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.974+0000 7f73e9187740 1 -- start start 2024-03-24T00:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.974+0000 7f73e9187740 1 Processor -- start 2024-03-24T00:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.975+0000 7f73e9187740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.975+0000 7f73e9187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.975+0000 7f73df400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73df400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:42114/0 (socket says 172.21.15.123:42114) 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73df400640 1 -- 172.21.15.123:0/461871341 learned_addr learned my addr 172.21.15.123:0/461871341 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32350dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0072830 con 0x55c3234f2f50 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0105550 con 0x55c3234edde0 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0072e90 con 0x55c3234ed3f0 2024-03-24T00:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73e9187740 10 client.0 _ll_register_callbacks cb 0x7f73cc009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73df400640 1 -- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 msgr2=0x55c32350fb50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73df400640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73df400640 1 -- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 msgr2=0x55c32350dcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.976+0000 7f73dfe00640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32350dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73df400640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32350dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73df400640 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c323507050 con 0x55c3234ed3f0 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73dfe00640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32350dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73df400640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73d00092a0 tx=0x7f73d003a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e876de620631919 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73c9600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d005a070 con 0x55c3234ed3f0 2024-03-24T00:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.977+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73d0004d90 con 0x55c3234ed3f0 2024-03-24T00:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.978+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d002cc00 con 0x55c3234ed3f0 2024-03-24T00:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.978+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d0004f90 con 0x55c3234ed3f0 2024-03-24T00:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.978+0000 7f73e9187740 10 client.25711 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.978+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c323570630 con 0x55c3234ed3f0 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d002cc00 con 0x55c3234ed3f0 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f73d0003c80 con 0x55c3234ed3f0 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 10 client.25711 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 10 client.25711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 20 client.25711 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c323570ec0 0x55c3235732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 4 client.25711 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.979+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c323508380 con 0x55c323570ec0 2024-03-24T00:40:11.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.983+0000 7f73dea00640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c323570ec0 0x55c3235732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:11.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.983+0000 7f73dea00640 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c323570ec0 0x55c3235732e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2d2f2b44365769 server_cookie=32b640946bbb5fbe in_seq=0 out_seq=0 2024-03-24T00:40:11.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:11.984+0000 7f73c9600640 10 client.25711 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73c9600640 1 -- 172.21.15.123:0/461871341 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c323508380 con 0x55c323570ec0 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73c9600640 10 client.25711 handle_command_reply: tid=0 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 client.25711 shutdown 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 msgr2=0x55c32350f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73d00092a0 tx=0x7f73d003a1c0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 shutdown_connections 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c3234f2f50 0x55c32350dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3234edde0 0x55c32350fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c323570ec0 0x55c3235732e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 --2- 172.21.15.123:0/461871341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c3234ed3f0 0x55c32350f610 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73c9600640 0 client.25711 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.115+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 >> 172.21.15.123:0/461871341 conn(0x55c3234e8960 msgr2=0x55c3234e72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 shutdown_connections 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e9187740 1 -- 172.21.15.123:0/461871341 wait complete. 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e9187740 20 client.25711 trim_cache size 0 max 16384 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:40:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 msgr2=0x7f73c40a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 0x7f73c40a4d70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f73d0036040 tx=0x7f73d0000bc0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 msgr2=0x7f73e019e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.116+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00d950 tx=0x7f73cc00de10 comp rx=0 tx=0).stop 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 shutdown_connections 2024-03-24T00:40:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f73c40a28b0 0x7f73c40a4d70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73e010dae0 0x7f73e01a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e00756e0 0x7f73e019e360 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 --2- 172.21.15.123:0/599166108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73e0074100 0x7f73e019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 >> 172.21.15.123:0/599166108 conn(0x7f73e00fca80 msgr2=0x7f73e0100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 shutdown_connections 2024-03-24T00:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:12.117+0000 7f73e4e00640 1 -- 172.21.15.123:0/599166108 wait complete. 2024-03-24T00:40:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:12 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:12 smithi204 ceph-mon[36345]: pgmap v454: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 283 MiB/s wr, 125 op/s 2024-03-24T00:40:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:12 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:40:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:12 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:12 smithi123 ceph-mon[31487]: pgmap v454: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 283 MiB/s wr, 125 op/s 2024-03-24T00:40:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:12 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:40:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:12 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:12 smithi177 ceph-mon[36034]: pgmap v454: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 283 MiB/s wr, 125 op/s 2024-03-24T00:40:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:12 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:40:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:14 smithi204 ceph-mon[36345]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 284 MiB/s wr, 119 op/s 2024-03-24T00:40:14.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:14 smithi123 ceph-mon[31487]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 284 MiB/s wr, 119 op/s 2024-03-24T00:40:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:14 smithi177 ceph-mon[36034]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 284 MiB/s wr, 119 op/s 2024-03-24T00:40:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:16 smithi123 ceph-mon[31487]: pgmap v456: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 284 MiB/s wr, 127 op/s 2024-03-24T00:40:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:16 smithi177 ceph-mon[36034]: pgmap v456: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 284 MiB/s wr, 127 op/s 2024-03-24T00:40:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:16 smithi204 ceph-mon[36345]: pgmap v456: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 284 MiB/s wr, 127 op/s 2024-03-24T00:40:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:17 smithi177 ceph-mon[36034]: pgmap v457: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 234 MiB/s wr, 127 op/s 2024-03-24T00:40:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:17 smithi204 ceph-mon[36345]: pgmap v457: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 234 MiB/s wr, 127 op/s 2024-03-24T00:40:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:17 smithi123 ceph-mon[31487]: pgmap v457: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 234 MiB/s wr, 127 op/s 2024-03-24T00:40:20.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:20 smithi204 ceph-mon[36345]: pgmap v458: 97 pgs: 97 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 270 MiB/s wr, 129 op/s 2024-03-24T00:40:20.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:20 smithi123 ceph-mon[31487]: pgmap v458: 97 pgs: 97 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 270 MiB/s wr, 129 op/s 2024-03-24T00:40:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:20 smithi177 ceph-mon[36034]: pgmap v458: 97 pgs: 97 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 270 MiB/s wr, 129 op/s 2024-03-24T00:40:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:22 smithi204 ceph-mon[36345]: pgmap v459: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 285 MiB/s wr, 142 op/s 2024-03-24T00:40:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:22 smithi123 ceph-mon[31487]: pgmap v459: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 285 MiB/s wr, 142 op/s 2024-03-24T00:40:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:22 smithi177 ceph-mon[36034]: pgmap v459: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 285 MiB/s wr, 142 op/s 2024-03-24T00:40:24.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:24 smithi204 ceph-mon[36345]: pgmap v460: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 280 MiB/s wr, 127 op/s 2024-03-24T00:40:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:24 smithi123 ceph-mon[31487]: pgmap v460: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 280 MiB/s wr, 127 op/s 2024-03-24T00:40:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:24 smithi177 ceph-mon[36034]: pgmap v460: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 280 MiB/s wr, 127 op/s 2024-03-24T00:40:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.321+0000 7fa5f2600640 1 -- 172.21.15.123:0/4014714863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec0ffd60 msgr2=0x7fa5ec102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.321+0000 7fa5f2600640 1 --2- 172.21.15.123:0/4014714863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec102190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009900 tx=0x7fa5e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:40:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.322+0000 7fa5f2600640 1 -- 172.21.15.123:0/4014714863 shutdown_connections 2024-03-24T00:40:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.322+0000 7fa5f2600640 1 --2- 172.21.15.123:0/4014714863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec105080 0x7fa5ec105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.322+0000 7fa5f2600640 1 --2- 172.21.15.123:0/4014714863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.322+0000 7fa5f2600640 1 --2- 172.21.15.123:0/4014714863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec102190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.322+0000 7fa5f2600640 1 -- 172.21.15.123:0/4014714863 >> 172.21.15.123:0/4014714863 conn(0x7fa5ec0fb980 msgr2=0x7fa5ec0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.323+0000 7fa5f2600640 1 -- 172.21.15.123:0/4014714863 shutdown_connections 2024-03-24T00:40:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.323+0000 7fa5f2600640 1 -- 172.21.15.123:0/4014714863 wait complete. 2024-03-24T00:40:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.325+0000 7fa5f2600640 1 Processor -- start 2024-03-24T00:40:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.325+0000 7fa5f2600640 1 -- start start 2024-03-24T00:40:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.326+0000 7fa5f2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f2600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f2600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec113830 con 0x7fa5ec0ffd60 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f2600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1136b0 con 0x7fa5ec1026d0 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f2600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1139b0 con 0x7fa5ec105080 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49306/0 (socket says 172.21.15.123:49306) 2024-03-24T00:40:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.327+0000 7fa5f1c00640 1 -- 172.21.15.123:0/72605633 learned_addr learned my addr 172.21.15.123:0/72605633 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1200640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 -- 172.21.15.123:0/72605633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 msgr2=0x7fa5ec070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 -- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 msgr2=0x7fa5ec070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 -- 172.21.15.123:0/72605633 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec070be0 con 0x7fa5ec105080 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1200640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400b970 tx=0x7fa5e400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.328+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.329+0000 7fa5e9600640 1 -- 172.21.15.123:0/72605633 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4019070 con 0x7fa5ec105080 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.329+0000 7fa5e9600640 1 -- 172.21.15.123:0/72605633 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e400cb40 con 0x7fa5ec105080 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.329+0000 7fa5e9600640 1 -- 172.21.15.123:0/72605633 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40076e0 con 0x7fa5ec105080 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.329+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec1ae5a0 con 0x7fa5ec105080 2024-03-24T00:40:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.329+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec1aea10 con 0x7fa5ec105080 2024-03-24T00:40:25.333 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.332+0000 7fa5e9600640 1 -- 172.21.15.123:0/72605633 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa5e401c070 con 0x7fa5ec105080 2024-03-24T00:40:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.333+0000 7fa5e9600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 0x7fa5d40a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.334 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.334+0000 7fa5e9600640 1 -- 172.21.15.123:0/72605633 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa5e40cdff0 con 0x7fa5ec105080 2024-03-24T00:40:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.334+0000 7fa5f1200640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 0x7fa5d40a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.335 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.335+0000 7fa5f1200640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 0x7fa5d40a4d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002c710 tx=0x7fa5e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.342 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.341+0000 7fa5f6a19740 1 Processor -- start 2024-03-24T00:40:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.345+0000 7fa5f6a19740 1 -- start start 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfbd13b0 con 0x5592bfc0ede0 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfb73830 con 0x5592bfc13f50 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfb1c5d0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc0e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc0e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49316/0 (socket says 172.21.15.123:49316) 2024-03-24T00:40:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.346+0000 7fa5f1200640 1 -- 172.21.15.123:0/2423543907 learned_addr learned my addr 172.21.15.123:0/2423543907 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 -- 172.21.15.123:0/2423543907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 msgr2=0x5592bfc0e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc0e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 -- 172.21.15.123:0/2423543907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 msgr2=0x5592bfc138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 -- 172.21.15.123:0/2423543907 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592bfb342c0 con 0x5592bfc13f50 2024-03-24T00:40:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.347+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc163e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400b940 tx=0x7fa5e4004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d488ced77179b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.348+0000 7fa5de000640 1 -- 172.21.15.123:0/2423543907 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40a3140 con 0x5592bfc13f50 2024-03-24T00:40:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.348+0000 7fa5de000640 1 -- 172.21.15.123:0/2423543907 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4098ca0 con 0x5592bfc13f50 2024-03-24T00:40:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.348+0000 7fa5de000640 1 -- 172.21.15.123:0/2423543907 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4099df0 con 0x5592bfc13f50 2024-03-24T00:40:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.349+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2423543907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 msgr2=0x5592bfc163e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.349+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc163e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400b940 tx=0x7fa5e4004b50 comp rx=0 tx=0).stop 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2423543907 shutdown_connections 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc163e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2423543907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2423543907 >> 172.21.15.123:0/2423543907 conn(0x5592bfc09960 msgr2=0x5592bfc09d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2423543907 shutdown_connections 2024-03-24T00:40:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.350+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2423543907 wait complete. 2024-03-24T00:40:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.351+0000 7fa5f6a19740 1 -- start start 2024-03-24T00:40:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.351+0000 7fa5f6a19740 1 Processor -- start 2024-03-24T00:40:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.352+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.352+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc18b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49328/0 (socket says 172.21.15.123:49328) 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc1fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfb2fa20 con 0x5592bfc0ede0 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfb6f8d0 con 0x5592bfc13f50 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.353+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592bfc16f00 con 0x5592bfc0e3f0 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 -- 172.21.15.123:0/410886647 learned_addr learned my addr 172.21.15.123:0/410886647 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc18b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f6a19740 10 client.0 _ll_register_callbacks cb 0x5592bfb6c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc1fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 -- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 msgr2=0x5592bfc1fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc1fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 -- 172.21.15.123:0/410886647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 msgr2=0x5592bfc18b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc18b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.354+0000 7fa5f1200640 1 -- 172.21.15.123:0/410886647 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592bf888210 con 0x5592bfc0e3f0 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.355+0000 7fa5f1200640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009850 tx=0x7fa5e0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a67dc1f47a9ce625 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.355+0000 7fa5d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.355+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00048d0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.355+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa5e0004a70 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5e4098ca0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e00056d0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5f6a19740 10 client.25723 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5592bf954620 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4098ca0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.356+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa5e0058510 con 0x5592bfc0e3f0 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 10 client.25723 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 10 client.25723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 20 client.25723 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc244e0 0x5592bfc268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 4 client.25723 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.357+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592bfb80d70 con 0x5592bfc244e0 2024-03-24T00:40:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.358+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc244e0 0x5592bfc268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.360+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc244e0 0x5592bfc268d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b29df67976cc3007 server_cookie=91112630b5823bdd in_seq=0 out_seq=0 2024-03-24T00:40:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.360+0000 7fa5d2a00640 10 client.25723 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.361+0000 7fa5d2a00640 1 -- 172.21.15.123:0/410886647 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5592bfb80d70 con 0x5592bfc244e0 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.361+0000 7fa5d2a00640 10 client.25723 handle_command_reply: tid=0 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.361+0000 7fa5f6a19740 1 client.25723 shutdown 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.361+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 msgr2=0x5592bfc18630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.361+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009850 tx=0x7fa5e0009260 comp rx=0 tx=0).stop 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 shutdown_connections 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc13f50 0x5592bfc1fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0ede0 0x5592bfc18b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc244e0 0x5592bfc268d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/410886647 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18630 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 >> 172.21.15.123:0/410886647 conn(0x5592bfc09960 msgr2=0x5592bfc08b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5d2a00640 0 client.25723 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 shutdown_connections 2024-03-24T00:40:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 1 -- 172.21.15.123:0/410886647 wait complete. 2024-03-24T00:40:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.362+0000 7fa5f6a19740 20 client.25723 trim_cache size 0 max 16384 2024-03-24T00:40:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.379+0000 7fa5f6a19740 1 Processor -- start 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.379+0000 7fa5f6a19740 1 -- start start 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0ede0 0x5592bfc1fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b5720 con 0x5592bfc0e3f0 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b7ce0 con 0x5592bfc0ede0 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b5d20 con 0x5592bfc13f50 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49336/0 (socket says 172.21.15.123:49336) 2024-03-24T00:40:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f1c00640 1 -- 172.21.15.123:0/3155088763 learned_addr learned my addr 172.21.15.123:0/3155088763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0ede0 0x5592bfc1fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.380+0000 7fa5f1200640 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 -- 172.21.15.123:0/3155088763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0ede0 msgr2=0x5592bfc1fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0ede0 0x5592bfc1fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 -- 172.21.15.123:0/3155088763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0e3f0 msgr2=0x5592bfc18ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 -- 172.21.15.123:0/3155088763 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592bfc26c40 con 0x5592bfc13f50 2024-03-24T00:40:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.381+0000 7fa5f1c00640 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4096e00 tx=0x7fa5e400bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b93b1069c47bc7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.383+0000 7fa5dd600640 1 -- 172.21.15.123:0/3155088763 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40b2020 con 0x5592bfc13f50 2024-03-24T00:40:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.383+0000 7fa5dd600640 1 -- 172.21.15.123:0/3155088763 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4098ca0 con 0x5592bfc13f50 2024-03-24T00:40:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.383+0000 7fa5dd600640 1 -- 172.21.15.123:0/3155088763 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4099df0 con 0x5592bfc13f50 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.384+0000 7fa5f6a19740 1 -- 172.21.15.123:0/3155088763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 msgr2=0x5592bfc0d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.384+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4096e00 tx=0x7fa5e400bf60 comp rx=0 tx=0).stop 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.384+0000 7fa5f6a19740 1 -- 172.21.15.123:0/3155088763 shutdown_connections 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.384+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc13f50 0x5592bfc0d2d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.384+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0ede0 0x5592bfc1fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.385+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/3155088763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc0e3f0 0x5592bfc18ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.385+0000 7fa5f6a19740 1 -- 172.21.15.123:0/3155088763 >> 172.21.15.123:0/3155088763 conn(0x5592bfc09960 msgr2=0x5592bfc0cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.385+0000 7fa5f6a19740 1 -- 172.21.15.123:0/3155088763 shutdown_connections 2024-03-24T00:40:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.386+0000 7fa5f6a19740 1 -- 172.21.15.123:0/3155088763 wait complete. 2024-03-24T00:40:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.388+0000 7fa5f6a19740 1 -- start start 2024-03-24T00:40:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.388+0000 7fa5f6a19740 1 Processor -- start 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.388+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 0x5592bfc1bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 0x5592bfc1c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc13f50 0x5592bfc1f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b6360 con 0x5592bfc13f50 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b7680 con 0x5592bfc0e3f0 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1b69c0 con 0x5592bfc0ede0 2024-03-24T00:40:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5f6a19740 10 client.0 _ll_register_callbacks cb 0x7fa5e002cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.389+0000 7fa5ebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 0x5592bfc1c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 0x5592bfc1bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5ebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 0x5592bfc1c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49340/0 (socket says 172.21.15.123:49340) 2024-03-24T00:40:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5ebe00640 1 -- 172.21.15.123:0/2262863365 learned_addr learned my addr 172.21.15.123:0/2262863365 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc13f50 0x5592bfc1f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 -- 172.21.15.123:0/2262863365 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 msgr2=0x5592bfc1c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 0x5592bfc1c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 -- 172.21.15.123:0/2262863365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc13f50 msgr2=0x5592bfc1f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc13f50 0x5592bfc1f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.390+0000 7fa5f1200640 1 -- 172.21.15.123:0/2262863365 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592bfc1f880 con 0x5592bfc0e3f0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.391+0000 7fa5f1200640 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 0x5592bfc1bd30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00588d0 tx=0x7fa5e00043d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1015d3a1d22abf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.391+0000 7fa5d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.391+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00048d0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.391+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa5e0004a70 con 0x5592bfc0e3f0 2024-03-24T00:40:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.391+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5e4098ca0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.392+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e0004c70 con 0x5592bfc0e3f0 2024-03-24T00:40:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.392+0000 7fa5f6a19740 10 client.25727 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.392+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5592bfc918c0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.392+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4098ca0 con 0x5592bfc0e3f0 2024-03-24T00:40:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.392+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa5e0069080 con 0x5592bfc0e3f0 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 10 client.25727 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 10 client.25727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 20 client.25727 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc92100 0x5592bfc94520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 4 client.25727 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5592bfb7b0c0 con 0x5592bfc92100 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.393+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc92100 0x5592bfc94520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.394+0000 7fa5ebe00640 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc92100 0x5592bfc94520 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42f8a43a981ec26 server_cookie=da420737e0cb2e8e in_seq=0 out_seq=0 2024-03-24T00:40:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.394+0000 7fa5d2a00640 10 client.25727 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.449+0000 7fa5d2a00640 1 -- 172.21.15.123:0/2262863365 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5592bfb7b0c0 con 0x5592bfc92100 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.449+0000 7fa5d2a00640 10 client.25727 handle_command_reply: tid=0 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.449+0000 7fa5f6a19740 1 client.25727 shutdown 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.449+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 msgr2=0x5592bfc1bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.449+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 0x5592bfc1bd30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00588d0 tx=0x7fa5e00043d0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 shutdown_connections 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5592bfc13f50 0x5592bfc1f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5592bfc92100 0x5592bfc94520 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5d2a00640 0 client.25727 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592bfc0ede0 0x5592bfc1c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 --2- 172.21.15.123:0/2262863365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592bfc0e3f0 0x5592bfc1bd30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 >> 172.21.15.123:0/2262863365 conn(0x5592bfc09960 msgr2=0x5592bfc188c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 shutdown_connections 2024-03-24T00:40:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 1 -- 172.21.15.123:0/2262863365 wait complete. 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.450+0000 7fa5f6a19740 20 client.25727 trim_cache size 0 max 16384 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.451+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 msgr2=0x7fa5d40a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.451+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 0x7fa5d40a4d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002c710 tx=0x7fa5e0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.451+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 msgr2=0x7fa5ec1adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.451+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400b970 tx=0x7fa5e400be30 comp rx=0 tx=0).stop 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 shutdown_connections 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5ec105080 0x7fa5ec1adf40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5ec1026d0 0x7fa5ec070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa5d40a28b0 0x7fa5d40a4d70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 --2- 172.21.15.123:0/72605633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0ffd60 0x7fa5ec070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.452+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 >> 172.21.15.123:0/72605633 conn(0x7fa5ec0fb980 msgr2=0x7fa5ec103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.453+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 shutdown_connections 2024-03-24T00:40:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:25.453+0000 7fa5f2600640 1 -- 172.21.15.123:0/72605633 wait complete. 2024-03-24T00:40:26.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:26 smithi204 ceph-mon[36345]: pgmap v461: 97 pgs: 97 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 285 MiB/s wr, 122 op/s 2024-03-24T00:40:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:26 smithi123 ceph-mon[31487]: pgmap v461: 97 pgs: 97 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 285 MiB/s wr, 122 op/s 2024-03-24T00:40:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:26 smithi177 ceph-mon[36034]: pgmap v461: 97 pgs: 97 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 285 MiB/s wr, 122 op/s 2024-03-24T00:40:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:28 smithi204 ceph-mon[36345]: pgmap v462: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 248 MiB/s wr, 98 op/s 2024-03-24T00:40:28.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:28 smithi123 ceph-mon[31487]: pgmap v462: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 248 MiB/s wr, 98 op/s 2024-03-24T00:40:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:28 smithi177 ceph-mon[36034]: pgmap v462: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 248 MiB/s wr, 98 op/s 2024-03-24T00:40:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:30 smithi204 ceph-mon[36345]: pgmap v463: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 282 MiB/s wr, 99 op/s 2024-03-24T00:40:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:30 smithi123 ceph-mon[31487]: pgmap v463: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 282 MiB/s wr, 99 op/s 2024-03-24T00:40:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:30 smithi177 ceph-mon[36034]: pgmap v463: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 282 MiB/s wr, 99 op/s 2024-03-24T00:40:30.938 INFO:tasks.workunit.client.0.smithi123.stdout:fs setup took 42 secs 2024-03-24T00:40:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:32 smithi204 ceph-mon[36345]: pgmap v464: 97 pgs: 97 active+clean; 9.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 295 MiB/s wr, 118 op/s 2024-03-24T00:40:32.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:32 smithi123 ceph-mon[31487]: pgmap v464: 97 pgs: 97 active+clean; 9.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 295 MiB/s wr, 118 op/s 2024-03-24T00:40:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:32 smithi177 ceph-mon[36034]: pgmap v464: 97 pgs: 97 active+clean; 9.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 295 MiB/s wr, 118 op/s 2024-03-24T00:40:34.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:34 smithi123 ceph-mon[31487]: pgmap v465: 97 pgs: 97 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 286 MiB/s wr, 106 op/s 2024-03-24T00:40:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:34 smithi177 ceph-mon[36034]: pgmap v465: 97 pgs: 97 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 286 MiB/s wr, 106 op/s 2024-03-24T00:40:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:34 smithi204 ceph-mon[36345]: pgmap v465: 97 pgs: 97 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 286 MiB/s wr, 106 op/s 2024-03-24T00:40:36.610 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:36 smithi123 ceph-mon[31487]: pgmap v466: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 324 MiB/s wr, 119 op/s 2024-03-24T00:40:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:36 smithi177 ceph-mon[36034]: pgmap v466: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 324 MiB/s wr, 119 op/s 2024-03-24T00:40:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:36 smithi204 ceph-mon[36345]: pgmap v466: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 324 MiB/s wr, 119 op/s 2024-03-24T00:40:37.543 INFO:tasks.workunit.client.0.smithi123.stdout:Syncing()...6 sec 2024-03-24T00:40:37.545 INFO:tasks.workunit.client.0.smithi123.stdout:Starting Actual Benchmark At: Sun Mar 24 00:40:37 2024 2024-03-24T00:40:37.545 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:40:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:37 smithi177 ceph-mon[36034]: pgmap v467: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 289 MiB/s wr, 101 op/s 2024-03-24T00:40:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:37 smithi204 ceph-mon[36345]: pgmap v467: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 289 MiB/s wr, 101 op/s 2024-03-24T00:40:37.871 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:37 smithi123 ceph-mon[31487]: pgmap v467: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 289 MiB/s wr, 101 op/s 2024-03-24T00:40:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.274+0000 7f69d5000640 1 -- 172.21.15.123:0/335523586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 msgr2=0x7f69d010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.274+0000 7f69d5000640 1 --2- 172.21.15.123:0/335523586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d010d480 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009900 tx=0x7f69bc02bc40 comp rx=0 tx=0).stop 2024-03-24T00:40:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.303+0000 7f69d5000640 1 -- 172.21.15.123:0/335523586 shutdown_connections 2024-03-24T00:40:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.303+0000 7f69d5000640 1 --2- 172.21.15.123:0/335523586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 0x7f69d010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.303+0000 7f69d5000640 1 --2- 172.21.15.123:0/335523586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d010d480 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.303+0000 7f69d5000640 1 --2- 172.21.15.123:0/335523586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d01086a0 0x7f69d010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.303+0000 7f69d5000640 1 -- 172.21.15.123:0/335523586 >> 172.21.15.123:0/335523586 conn(0x7f69d006e6e0 msgr2=0x7f69d0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.306+0000 7f69d5000640 1 -- 172.21.15.123:0/335523586 shutdown_connections 2024-03-24T00:40:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.307+0000 7f69d5000640 1 -- 172.21.15.123:0/335523586 wait complete. 2024-03-24T00:40:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.308+0000 7f69d5000640 1 Processor -- start 2024-03-24T00:40:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.318+0000 7f69d5000640 1 -- start start 2024-03-24T00:40:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.318+0000 7f69d5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d01086a0 0x7f69d0114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.324+0000 7f69d5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.324+0000 7f69d5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 0x7f69d01b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.324+0000 7f69d5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d011c1d0 con 0x7f69d01086a0 2024-03-24T00:40:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.324+0000 7f69d5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d011c050 con 0x7f69d010d9c0 2024-03-24T00:40:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.324+0000 7f69d5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d011c350 con 0x7f69d010b010 2024-03-24T00:40:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.326+0000 7f69cd400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.326+0000 7f69cd400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43244/0 (socket says 172.21.15.123:43244) 2024-03-24T00:40:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.326+0000 7f69cd400640 1 -- 172.21.15.123:0/1032281973 learned_addr learned my addr 172.21.15.123:0/1032281973 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.327+0000 7f69ce800640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 0x7f69d01b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.327+0000 7f69cd400640 1 -- 172.21.15.123:0/1032281973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 msgr2=0x7f69d01b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.327+0000 7f69cd400640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 0x7f69d01b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.327+0000 7f69cd400640 1 -- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d01086a0 msgr2=0x7f69d0114c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.328+0000 7f69cd400640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d01086a0 0x7f69d0114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.328+0000 7f69cd400640 1 -- 172.21.15.123:0/1032281973 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0118330 con 0x7f69d010b010 2024-03-24T00:40:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.330+0000 7f69cd400640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69bc02c8f0 tx=0x7f69bc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.330+0000 7f69c6a00640 1 -- 172.21.15.123:0/1032281973 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc041070 con 0x7f69d010b010 2024-03-24T00:40:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.346+0000 7f69c6a00640 1 -- 172.21.15.123:0/1032281973 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69bc0044d0 con 0x7f69d010b010 2024-03-24T00:40:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.346+0000 7f69c6a00640 1 -- 172.21.15.123:0/1032281973 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc041070 con 0x7f69d010b010 2024-03-24T00:40:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.349+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d01b6e20 con 0x7f69d010b010 2024-03-24T00:40:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.349+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d01b7230 con 0x7f69d010b010 2024-03-24T00:40:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.395+0000 7f69d7daf740 1 Processor -- start 2024-03-24T00:40:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.406+0000 7f69d7daf740 1 -- start start 2024-03-24T00:40:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.409+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.410+0000 7f69ce800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.410+0000 7f69ce800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43260/0 (socket says 172.21.15.123:43260) 2024-03-24T00:40:39.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.410+0000 7f69ce800640 1 -- 172.21.15.123:0/2297027097 learned_addr learned my addr 172.21.15.123:0/2297027097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:39.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.411+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 0x564002a296d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.412+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.412+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640029e73b0 con 0x564002a29d40 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.412+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564002989830 con 0x564002a24bd0 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.412+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640029325d0 con 0x564002a241e0 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.412+0000 7f69cde00640 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 0x564002a296d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 -- 172.21.15.123:0/2297027097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 msgr2=0x564002a24600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 -- 172.21.15.123:0/2297027097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 msgr2=0x564002a2c1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:40:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2c1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 -- 172.21.15.123:0/2297027097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56400294a2c0 con 0x564002a24bd0 2024-03-24T00:40:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69ce800640 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.413+0000 7f69cde00640 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 0x564002a296d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009750 tx=0x7f69c0009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=633b4db707eac9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.414+0000 7f69c6a00640 1 -- 172.21.15.123:0/1032281973 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f69bc044070 con 0x7f69d010b010 2024-03-24T00:40:39.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.416+0000 7f69c6a00640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 0x7f69b40a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.417+0000 7f69ce800640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 0x7f69b40a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.446+0000 7f69c6a00640 1 -- 172.21.15.123:0/1032281973 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69bc0efeb0 con 0x7f69d010b010 2024-03-24T00:40:39.454 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.447+0000 7f69ce800640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 0x7f69b40a5070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x564002a25430 tx=0x7f69c8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.448+0000 7f69bb400640 1 -- 172.21.15.123:0/2297027097 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0015020 con 0x564002a24bd0 2024-03-24T00:40:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.448+0000 7f69bb400640 1 -- 172.21.15.123:0/2297027097 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69c0011420 con 0x564002a24bd0 2024-03-24T00:40:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.448+0000 7f69bb400640 1 -- 172.21.15.123:0/2297027097 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0009e60 con 0x564002a24bd0 2024-03-24T00:40:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.472+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 msgr2=0x564002a296d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.472+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 0x564002a296d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009750 tx=0x7f69c0009c10 comp rx=0 tx=0).stop 2024-03-24T00:40:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.475+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 shutdown_connections 2024-03-24T00:40:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.475+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.475+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a24bd0 0x564002a296d0 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009750 tx=0x7f69c0009c10 comp rx=0 tx=0).stop 2024-03-24T00:40:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.475+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2297027097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a24600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.475+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 >> 172.21.15.123:0/2297027097 conn(0x564002a1f750 msgr2=0x564002a1fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.491 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.490+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 shutdown_connections 2024-03-24T00:40:39.491 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.490+0000 7f69d7daf740 1 -- 172.21.15.123:0/2297027097 wait complete. 2024-03-24T00:40:39.495 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.492+0000 7f69d7daf740 1 -- start start 2024-03-24T00:40:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.496+0000 7f69d7daf740 1 Processor -- start 2024-03-24T00:40:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.496+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.501+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.501+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 0x564002a35f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.501+0000 7f69cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.502+0000 7f69cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43196/0 (socket says 172.21.15.123:43196) 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.502+0000 7f69cde00640 1 -- 172.21.15.123:0/1400466386 learned_addr learned my addr 172.21.15.123:0/1400466386 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.502+0000 7f69ce800640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 0x564002a35f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.502+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564002945a20 con 0x564002a29d40 2024-03-24T00:40:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.504+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640029858d0 con 0x564002a241e0 2024-03-24T00:40:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.504+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564002a2ccf0 con 0x564002a33b10 2024-03-24T00:40:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.504+0000 7f69d7daf740 10 client.0 _ll_register_callbacks cb 0x5640029821c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.504+0000 7f69cd400640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.505+0000 7f69cd400640 1 -- 172.21.15.123:0/1400466386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 msgr2=0x564002a35f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.505+0000 7f69cd400640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 0x564002a35f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.505+0000 7f69cd400640 1 -- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 msgr2=0x564002a2e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.505+0000 7f69cd400640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.505+0000 7f69cd400640 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56400269e210 con 0x564002a29d40 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.507+0000 7f69cde00640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.507+0000 7f69ce800640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 0x564002a35f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.507+0000 7f69cd400640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2eae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69bc006010 tx=0x7f69bc0bc160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5defba10d5d33d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.507+0000 7f69afe00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.528+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0d5060 con 0x564002a29d40 2024-03-24T00:40:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.528+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69bc0b8e70 con 0x564002a29d40 2024-03-24T00:40:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.544+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69c0011420 con 0x564002a29d40 2024-03-24T00:40:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.547+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69bc0da050 con 0x564002a29d40 2024-03-24T00:40:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.552+0000 7f69d7daf740 10 client.15606 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.552+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56400276a620 con 0x564002a29d40 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69c0011420 con 0x564002a29d40 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f69bc0d0be0 con 0x564002a29d40 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 10 client.15606 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 20 client.15606 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a3a6a0 0x564002a3ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 4 client.15606 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.563+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564002996d70 con 0x564002a3a6a0 2024-03-24T00:40:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.565+0000 7f69cde00640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a3a6a0 0x564002a3ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.571+0000 7f69cde00640 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a3a6a0 0x564002a3ca90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7474579732a0acf server_cookie=d05ad53eb56405c6 in_seq=0 out_seq=0 2024-03-24T00:40:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.571+0000 7f69afe00640 10 client.15606 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.577+0000 7f69afe00640 1 -- 172.21.15.123:0/1400466386 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564002996d70 con 0x564002a3a6a0 2024-03-24T00:40:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.577+0000 7f69afe00640 10 client.15606 handle_command_reply: tid=0 2024-03-24T00:40:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.577+0000 7f69d7daf740 1 client.15606 shutdown 2024-03-24T00:40:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.577+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 msgr2=0x564002a2eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.577+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2eae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69bc006010 tx=0x7f69bc0bc160 comp rx=0 tx=0).stop 2024-03-24T00:40:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 shutdown_connections 2024-03-24T00:40:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a3a6a0 0x564002a3ca90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a33b10 0x564002a35f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a29d40 0x564002a2eae0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 --2- 172.21.15.123:0/1400466386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a241e0 0x564002a2e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 >> 172.21.15.123:0/1400466386 conn(0x564002a1f750 msgr2=0x564002a1e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69afe00640 0 client.15606 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 shutdown_connections 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.578+0000 7f69d7daf740 1 -- 172.21.15.123:0/1400466386 wait complete. 2024-03-24T00:40:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.580+0000 7f69d7daf740 20 client.15606 trim_cache size 0 max 16384 2024-03-24T00:40:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.617+0000 7f69d7daf740 1 Processor -- start 2024-03-24T00:40:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.617+0000 7f69d7daf740 1 -- start start 2024-03-24T00:40:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.618+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 0x564002a260d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.620 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.619+0000 7f69cde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 0x564002a260d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.620 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.619+0000 7f69cde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 0x564002a260d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53904/0 (socket says 172.21.15.123:53904) 2024-03-24T00:40:39.620 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.620+0000 7f69cde00640 1 -- 172.21.15.123:0/3232360671 learned_addr learned my addr 172.21.15.123:0/3232360671 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:39.620 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.620+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 0x564002a26610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.621 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.620+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a33b10 0x564002a1e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.621 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.620+0000 7f69cd400640 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 0x564002a26610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.621 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.621+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01be170 con 0x564002a241e0 2024-03-24T00:40:39.622 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.621+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01c0730 con 0x564002a33b10 2024-03-24T00:40:39.622 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.621+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01be770 con 0x564002a29d40 2024-03-24T00:40:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.621+0000 7f69ce800640 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a33b10 0x564002a1e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.622+0000 7f69cd400640 1 -- 172.21.15.123:0/3232360671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a33b10 msgr2=0x564002a1e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.622+0000 7f69cd400640 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a33b10 0x564002a1e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.622+0000 7f69cd400640 1 -- 172.21.15.123:0/3232360671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 msgr2=0x564002a260d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.622+0000 7f69cd400640 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 0x564002a260d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.622+0000 7f69cd400640 1 -- 172.21.15.123:0/3232360671 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564002a3ce00 con 0x564002a29d40 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.623+0000 7f69cd400640 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 0x564002a26610 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69bc049000 tx=0x7f69bc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce47a05e47c81ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.623+0000 7f69baa00640 1 -- 172.21.15.123:0/3232360671 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0d5070 con 0x564002a29d40 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.624+0000 7f69baa00640 1 -- 172.21.15.123:0/3232360671 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69bc0c4d50 con 0x564002a29d40 2024-03-24T00:40:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.624+0000 7f69baa00640 1 -- 172.21.15.123:0/3232360671 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0a70b0 con 0x564002a29d40 2024-03-24T00:40:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.625+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 msgr2=0x564002a26610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.625+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 0x564002a26610 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69bc049000 tx=0x7f69bc009260 comp rx=0 tx=0).stop 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.625+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 shutdown_connections 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.625+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a33b10 0x564002a1e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.626+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a29d40 0x564002a26610 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.626+0000 7f69d7daf740 1 --2- 172.21.15.123:0/3232360671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a241e0 0x564002a260d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.626+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 >> 172.21.15.123:0/3232360671 conn(0x564002a1f750 msgr2=0x564002a2e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.626+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 shutdown_connections 2024-03-24T00:40:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.626+0000 7f69d7daf740 1 -- 172.21.15.123:0/3232360671 wait complete. 2024-03-24T00:40:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.628+0000 7f69d7daf740 1 -- start start 2024-03-24T00:40:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.628+0000 7f69d7daf740 1 Processor -- start 2024-03-24T00:40:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.629+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a4b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.629+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.629+0000 7f69d7daf740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a33b10 0x564002a4be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.630+0000 7f69cde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a4b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.630+0000 7f69cde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a4b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43286/0 (socket says 172.21.15.123:43286) 2024-03-24T00:40:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.630+0000 7f69cd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.630+0000 7f69cd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43218/0 (socket says 172.21.15.123:43218) 2024-03-24T00:40:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.630+0000 7f69cde00640 1 -- 172.21.15.123:0/2039787887 learned_addr learned my addr 172.21.15.123:0/2039787887 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.631+0000 7f69ce800640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a33b10 0x564002a4be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.631+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01bedb0 con 0x564002a33b10 2024-03-24T00:40:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.631+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01c00d0 con 0x564002a29d40 2024-03-24T00:40:39.634 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.632+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01bf410 con 0x564002a241e0 2024-03-24T00:40:39.634 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.632+0000 7f69cd400640 1 -- 172.21.15.123:0/2039787887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 msgr2=0x564002a4b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.632+0000 7f69cd400640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a4b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.632+0000 7f69cd400640 1 -- 172.21.15.123:0/2039787887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a33b10 msgr2=0x564002a4be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.633+0000 7f69cd400640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a33b10 0x564002a4be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.633+0000 7f69cd400640 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564002a953d0 con 0x564002a29d40 2024-03-24T00:40:39.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.634+0000 7f69d7daf740 10 client.0 _ll_register_callbacks cb 0x7f69bc0bc5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:39.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.635+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69bc0c4d50 con 0x564002a29d40 2024-03-24T00:40:39.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.636+0000 7f69afe00640 10 client.25742 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.636+0000 7f69cd400640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0baab0 tx=0x7f69bc02cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3809bfc02ff1abed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.636+0000 7f69afe00640 1 -- 172.21.15.123:0/2039787887 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0d5040 con 0x564002a29d40 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.636+0000 7f69afe00640 1 -- 172.21.15.123:0/2039787887 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69bc0a0070 con 0x564002a29d40 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.636+0000 7f69afe00640 1 -- 172.21.15.123:0/2039787887 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69bc0d5040 con 0x564002a29d40 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.637+0000 7f69d7daf740 10 client.25742 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.637+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=19}) v3 -- 0x564002a953d0 con 0x564002a29d40 2024-03-24T00:40:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69afe00640 1 -- 172.21.15.123:0/2039787887 <== mon.2 v2:172.21.15.177:3300/0 4 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f69bc0035f0 con 0x564002a29d40 2024-03-24T00:40:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 10 client.25742 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 20 client.25742 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a96850 0x564002a98ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 4 client.25742 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:40:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.638+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5640029910c0 con 0x564002a96850 2024-03-24T00:40:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.641+0000 7f69cde00640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a96850 0x564002a98ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.642+0000 7f69cde00640 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a96850 0x564002a98ce0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=282669dbc346986e server_cookie=44f7bb6134801a61 in_seq=0 out_seq=0 2024-03-24T00:40:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:39.642+0000 7f69afe00640 10 client.25742 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:40.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:40 smithi204 ceph-mon[36345]: pgmap v468: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 348 MiB/s wr, 115 op/s 2024-03-24T00:40:40.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:40 smithi123 ceph-mon[31487]: pgmap v468: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 348 MiB/s wr, 115 op/s 2024-03-24T00:40:40.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:40 smithi177 ceph-mon[36034]: pgmap v468: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 348 MiB/s wr, 115 op/s 2024-03-24T00:40:41.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:40:41.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:40:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 -- 172.21.15.123:0/4033058874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281049c0 msgr2=0x7fe128106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 --2- 172.21.15.123:0/4033058874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281049c0 0x7fe128106db0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe114009900 tx=0x7fe11402bc30 comp rx=0 tx=0).stop 2024-03-24T00:40:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 -- 172.21.15.123:0/4033058874 shutdown_connections 2024-03-24T00:40:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 --2- 172.21.15.123:0/4033058874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281072f0 0x7fe1281096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 --2- 172.21.15.123:0/4033058874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281049c0 0x7fe128106db0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 --2- 172.21.15.123:0/4033058874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe128104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 -- 172.21.15.123:0/4033058874 >> 172.21.15.123:0/4033058874 conn(0x7fe1280fba20 msgr2=0x7fe1280fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 -- 172.21.15.123:0/4033058874 shutdown_connections 2024-03-24T00:40:41.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.581+0000 7fe127e00640 1 -- 172.21.15.123:0/4033058874 wait complete. 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 Processor -- start 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 -- start start 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe1281978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281049c0 0x7fe128197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810d270 con 0x7fe128102090 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810d0f0 con 0x7fe1281072f0 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.582+0000 7fe127e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810d3f0 con 0x7fe1281049c0 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe126a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe1281978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe126000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281049c0 0x7fe128197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43236/0 (socket says 172.21.15.123:43236) 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe126a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe1281978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53934/0 (socket says 172.21.15.123:53934) 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 -- 172.21.15.123:0/3979381808 learned_addr learned my addr 172.21.15.123:0/3979381808 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 -- 172.21.15.123:0/3979381808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281049c0 msgr2=0x7fe128197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281049c0 0x7fe128197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 -- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 msgr2=0x7fe1281978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe1281978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 -- 172.21.15.123:0/3979381808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12819f4e0 con 0x7fe1281072f0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.583+0000 7fe127400640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe12000eb10 tx=0x7fe12000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.584+0000 7fe11f400640 1 -- 172.21.15.123:0/3979381808 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120004b20 con 0x7fe1281072f0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.584+0000 7fe11f400640 1 -- 172.21.15.123:0/3979381808 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe120004cc0 con 0x7fe1281072f0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.584+0000 7fe11f400640 1 -- 172.21.15.123:0/3979381808 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120010760 con 0x7fe1281072f0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.584+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1280ffee0 con 0x7fe1281072f0 2024-03-24T00:40:41.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.584+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe128100400 con 0x7fe1281072f0 2024-03-24T00:40:41.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.587+0000 7fe13038c740 1 Processor -- start 2024-03-24T00:40:41.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.587+0000 7fe11f400640 1 -- 172.21.15.123:0/3979381808 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe120005020 con 0x7fe1281072f0 2024-03-24T00:40:41.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.587+0000 7fe13038c740 1 -- start start 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 0x5642cbdb0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb5bb0 0x5642cbdb8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbd15250 con 0x5642cbdb0050 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbcbda30 con 0x5642cbdb0a40 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe13038c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbd112f0 con 0x5642cbdb5bb0 2024-03-24T00:40:41.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 0x5642cbdb0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 0x5642cbdb0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53946/0 (socket says 172.21.15.123:53946) 2024-03-24T00:40:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:43238/0 (socket says 172.21.15.123:43238) 2024-03-24T00:40:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126000640 1 -- 172.21.15.123:0/3955360319 learned_addr learned my addr 172.21.15.123:0/3955360319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126000640 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb5bb0 0x5642cbdb8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 -- 172.21.15.123:0/3955360319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb5bb0 msgr2=0x5642cbdb8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb5bb0 0x5642cbdb8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 -- 172.21.15.123:0/3955360319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 msgr2=0x5642cbdb0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 0x5642cbdb0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.588+0000 7fe126a00640 1 -- 172.21.15.123:0/3955360319 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642cbca8690 con 0x5642cbdb0a40 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe126a00640 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe118005fd0 tx=0x7fe118009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae5061810523f248 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe11f400640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe113400640 1 -- 172.21.15.123:0/3955360319 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118004220 con 0x5642cbdb0a40 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe113400640 1 -- 172.21.15.123:0/3955360319 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe11800f4b0 con 0x5642cbdb0a40 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe113400640 1 -- 172.21.15.123:0/3955360319 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800d820 con 0x5642cbdb0a40 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe11f400640 1 -- 172.21.15.123:0/3979381808 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe1200cf070 con 0x7fe1281072f0 2024-03-24T00:40:41.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.589+0000 7fe126000640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 -- 172.21.15.123:0/3955360319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 msgr2=0x5642cbdb5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe118005fd0 tx=0x7fe118009780 comp rx=0 tx=0).stop 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe126000640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5642cbdb12a0 tx=0x7fe114005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 -- 172.21.15.123:0/3955360319 shutdown_connections 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb5bb0 0x5642cbdb8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdb5540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 --2- 172.21.15.123:0/3955360319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0050 0x5642cbdb0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 -- 172.21.15.123:0/3955360319 >> 172.21.15.123:0/3955360319 conn(0x5642cbd78200 msgr2=0x5642cbdab9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 -- 172.21.15.123:0/3955360319 shutdown_connections 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.590+0000 7fe13038c740 1 -- 172.21.15.123:0/3955360319 wait complete. 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.591+0000 7fe13038c740 1 -- start start 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.591+0000 7fe13038c740 1 Processor -- start 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0a40 0x5642cbdba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbdb8b60 con 0x5642cbdb0a40 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbcd0e80 con 0x5642cbdb5bb0 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642cbdb8ce0 con 0x5642cbdb0050 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe13038c740 10 client.0 _ll_register_callbacks cb 0x5642cbce3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43322/0 (socket says 172.21.15.123:43322) 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 -- 172.21.15.123:0/1849431140 learned_addr learned my addr 172.21.15.123:0/1849431140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126000640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0a40 0x5642cbdba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe127400640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 -- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 msgr2=0x5642cbdc15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 -- 172.21.15.123:0/1849431140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0a40 msgr2=0x5642cbdba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0a40 0x5642cbdba590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe126a00640 1 -- 172.21.15.123:0/1849431140 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642cbaf37d0 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.592+0000 7fe127400640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe126a00640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe118009c80 tx=0x7fe118003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed580812a7ad1cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe107e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800fcd0 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642cbd22790 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe118005050 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.593+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe118005250 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 10 client.25750 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5642cba293f0 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5642cbd22790 con 0x5642cbdb0050 2024-03-24T00:40:41.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe11800d580 con 0x5642cbdb0050 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 10 client.25750 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 10 client.25750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 20 client.25750 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbdc5ac0 0x5642cbdc7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 4 client.25750 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:41.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.594+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642cbd1cae0 con 0x5642cbdc5ac0 2024-03-24T00:40:41.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.595+0000 7fe126000640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbdc5ac0 0x5642cbdc7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.595+0000 7fe126000640 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbdc5ac0 0x5642cbdc7eb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6f743840b096cbc server_cookie=ad820954325f6d72 in_seq=0 out_seq=0 2024-03-24T00:40:41.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:41.595+0000 7fe107e00640 10 client.25750 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:42.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:42 smithi204 ceph-mon[36345]: pgmap v469: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 349 MiB/s wr, 862 op/s 2024-03-24T00:40:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:42 smithi123 ceph-mon[31487]: pgmap v469: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 349 MiB/s wr, 862 op/s 2024-03-24T00:40:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:42 smithi177 ceph-mon[36034]: pgmap v469: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 349 MiB/s wr, 862 op/s 2024-03-24T00:40:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:44 smithi123 ceph-mon[31487]: pgmap v470: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 285 MiB/s wr, 1.58k op/s 2024-03-24T00:40:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:44 smithi177 ceph-mon[36034]: pgmap v470: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 285 MiB/s wr, 1.58k op/s 2024-03-24T00:40:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:44 smithi204 ceph-mon[36345]: pgmap v470: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 285 MiB/s wr, 1.58k op/s 2024-03-24T00:40:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:46 smithi123 ceph-mon[31487]: pgmap v471: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 248 MiB/s wr, 3.09k op/s 2024-03-24T00:40:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:46 smithi177 ceph-mon[36034]: pgmap v471: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 248 MiB/s wr, 3.09k op/s 2024-03-24T00:40:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:46 smithi204 ceph-mon[36345]: pgmap v471: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 248 MiB/s wr, 3.09k op/s 2024-03-24T00:40:48.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:48 smithi123 ceph-mon[31487]: pgmap v472: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 174 MiB/s wr, 3.46k op/s 2024-03-24T00:40:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:48 smithi177 ceph-mon[36034]: pgmap v472: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 174 MiB/s wr, 3.46k op/s 2024-03-24T00:40:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:48 smithi204 ceph-mon[36345]: pgmap v472: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 174 MiB/s wr, 3.46k op/s 2024-03-24T00:40:50.609 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:50 smithi123 ceph-mon[31487]: pgmap v473: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 165 MiB/s wr, 4.99k op/s 2024-03-24T00:40:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:50 smithi177 ceph-mon[36034]: pgmap v473: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 165 MiB/s wr, 4.99k op/s 2024-03-24T00:40:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:50 smithi204 ceph-mon[36345]: pgmap v473: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 165 MiB/s wr, 4.99k op/s 2024-03-24T00:40:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:52 smithi123 ceph-mon[31487]: pgmap v474: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 99 MiB/s wr, 6.45k op/s 2024-03-24T00:40:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:52 smithi177 ceph-mon[36034]: pgmap v474: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 99 MiB/s wr, 6.45k op/s 2024-03-24T00:40:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:52 smithi204 ceph-mon[36345]: pgmap v474: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 99 MiB/s wr, 6.45k op/s 2024-03-24T00:40:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.496+0000 7f69afe00640 1 -- 172.21.15.123:0/2039787887 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5640029910c0 con 0x564002a96850 2024-03-24T00:40:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.496+0000 7f69afe00640 10 client.25742 handle_command_reply: tid=0 2024-03-24T00:40:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.496+0000 7f69d7daf740 1 client.25742 shutdown 2024-03-24T00:40:54.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.497+0000 7fe107e00640 1 -- 172.21.15.123:0/1849431140 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5642cbd1cae0 con 0x5642cbdc5ac0 2024-03-24T00:40:54.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.497+0000 7fe107e00640 10 client.25750 handle_command_reply: tid=0 2024-03-24T00:40:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 msgr2=0x564002a4b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0baab0 tx=0x7f69bc02cf90 comp rx=0 tx=0).stop 2024-03-24T00:40:54.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.497+0000 7fe13038c740 1 client.25750 shutdown 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 shutdown_connections 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564002a33b10 0x564002a4be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564002a29d40 0x564002a4b920 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564002a96850 0x564002a98ce0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.497+0000 7f69d7daf740 1 --2- 172.21.15.123:0/2039787887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564002a241e0 0x564002a4b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.498+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 >> 172.21.15.123:0/2039787887 conn(0x564002a1f750 msgr2=0x564002a2e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.497+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 msgr2=0x5642cbdba050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.497+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe118009c80 tx=0x7fe118003820 comp rx=0 tx=0).stop 2024-03-24T00:40:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.498+0000 7f69afe00640 0 client.25742 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.498+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 shutdown_connections 2024-03-24T00:40:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 shutdown_connections 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbdb0a40 0x5642cbdba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbdc5ac0 0x5642cbdc7eb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 --2- 172.21.15.123:0/1849431140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdba050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 >> 172.21.15.123:0/1849431140 conn(0x5642cbd78200 msgr2=0x5642cbdaabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe107e00640 0 client.25750 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.498+0000 7f69d7daf740 1 -- 172.21.15.123:0/2039787887 wait complete. 2024-03-24T00:40:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.498+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 shutdown_connections 2024-03-24T00:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.498+0000 7f69d7daf740 20 client.25742 trim_cache size 0 max 16384 2024-03-24T00:40:54.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.499+0000 7fe13038c740 1 -- 172.21.15.123:0/1849431140 wait complete. 2024-03-24T00:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:40:54.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.499+0000 7fe13038c740 20 client.25750 trim_cache size 0 max 16384 2024-03-24T00:40:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 msgr2=0x7f69b40a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 0x7f69b40a5070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x564002a25430 tx=0x7f69c8009500 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 msgr2=0x7f69d01151b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69bc02c8f0 tx=0x7f69bc005d50 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 shutdown_connections 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69d010d9c0 0x7f69d01b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69d010b010 0x7f69d01151b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f69b40a2bb0 0x7f69b40a5070 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 --2- 172.21.15.123:0/1032281973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d01086a0 0x7f69d0114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.500+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 >> 172.21.15.123:0/1032281973 conn(0x7f69d006e6e0 msgr2=0x7f69d010a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.501+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 shutdown_connections 2024-03-24T00:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:40:54.502+0000 7f69d5000640 1 -- 172.21.15.123:0/1032281973 wait complete. 2024-03-24T00:40:54.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.551+0000 7fe13038c740 1 Processor -- start 2024-03-24T00:40:54.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.551+0000 7fe13038c740 1 -- start start 2024-03-24T00:40:54.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdb1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdcc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a4820 con 0x5642cbd78200 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a6de0 con 0x5642cbdb0a40 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.552+0000 7fe13038c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a4e20 con 0x5642cbdb0050 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.554+0000 7fe126000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.554+0000 7fe126000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34846/0 (socket says 172.21.15.123:34846) 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.554+0000 7fe126000640 1 -- 172.21.15.123:0/3520157328 learned_addr learned my addr 172.21.15.123:0/3520157328 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.555+0000 7fe126000640 1 -- 172.21.15.123:0/3520157328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 msgr2=0x5642cbdcc6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.555+0000 7fe126000640 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdcc6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.555+0000 7fe126000640 1 -- 172.21.15.123:0/3520157328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 msgr2=0x5642cbdb1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.555+0000 7fe126000640 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdb1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.555+0000 7fe126000640 1 -- 172.21.15.123:0/3520157328 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642cb9e90f0 con 0x5642cbdb0050 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.556+0000 7fe126000640 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe114036040 tx=0x7fe114009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d762b77bc67bb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.557+0000 7fe112a00640 1 -- 172.21.15.123:0/3520157328 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114061070 con 0x5642cbdb0050 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.557+0000 7fe112a00640 1 -- 172.21.15.123:0/3520157328 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe114005040 con 0x5642cbdb0050 2024-03-24T00:40:54.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.557+0000 7fe112a00640 1 -- 172.21.15.123:0/3520157328 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140046c0 con 0x5642cbdb0050 2024-03-24T00:40:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.565+0000 7fe13038c740 1 -- 172.21.15.123:0/3520157328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 msgr2=0x5642cbdc1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.565+0000 7fe13038c740 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe114036040 tx=0x7fe114009260 comp rx=0 tx=0).stop 2024-03-24T00:40:54.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.566+0000 7fe13038c740 1 -- 172.21.15.123:0/3520157328 shutdown_connections 2024-03-24T00:40:54.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.566+0000 7fe13038c740 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb0a40 0x5642cbdcc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.566+0000 7fe13038c740 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0050 0x5642cbdc1590 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe114036040 tx=0x7fe114009260 comp rx=0 tx=0).stop 2024-03-24T00:40:54.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.566+0000 7fe13038c740 1 --2- 172.21.15.123:0/3520157328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdb1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.566+0000 7fe13038c740 1 -- 172.21.15.123:0/3520157328 >> 172.21.15.123:0/3520157328 conn(0x5642cbdc82a0 msgr2=0x5642cbdb14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.573+0000 7fe13038c740 1 -- 172.21.15.123:0/3520157328 shutdown_connections 2024-03-24T00:40:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.573+0000 7fe13038c740 1 -- 172.21.15.123:0/3520157328 wait complete. 2024-03-24T00:40:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.574+0000 7fe13038c740 1 -- start start 2024-03-24T00:40:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.574+0000 7fe13038c740 1 Processor -- start 2024-03-24T00:40:54.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.578+0000 7fe13038c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdbcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.578+0000 7fe13038c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0a40 0x5642cbdbd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.578+0000 7fe13038c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe13038c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a5460 con 0x5642cbd78200 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe13038c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a6780 con 0x5642cbdb5bb0 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe13038c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281a5ac0 con 0x5642cbdb0a40 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe127400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe127400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47080/0 (socket says 172.21.15.123:47080) 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe127400640 1 -- 172.21.15.123:0/3198540786 learned_addr learned my addr 172.21.15.123:0/3198540786 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.581+0000 7fe126a00640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdbcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.582+0000 7fe13038c740 10 client.0 _ll_register_callbacks cb 0x7fe114009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.582+0000 7fe126000640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0a40 0x5642cbdbd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.582+0000 7fe127400640 1 -- 172.21.15.123:0/3198540786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0a40 msgr2=0x5642cbdbd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.582+0000 7fe127400640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0a40 0x5642cbdbd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.582+0000 7fe127400640 1 -- 172.21.15.123:0/3198540786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 msgr2=0x5642cbdbcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:54.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.584+0000 7fe127400640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdbcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.584+0000 7fe127400640 1 -- 172.21.15.123:0/3198540786 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642cbdc08e0 con 0x5642cbdb5bb0 2024-03-24T00:40:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.584+0000 7fe126a00640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdbcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.584+0000 7fe107e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.585+0000 7fe127400640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe1200049d0 tx=0x7fe1200058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f99a57e7a19cf33b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.586+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12009c120 con 0x5642cbdb5bb0 2024-03-24T00:40:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.587+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe1200afe00 con 0x5642cbdb5bb0 2024-03-24T00:40:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.587+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe114005040 con 0x5642cbdb5bb0 2024-03-24T00:40:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.588+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12009ba20 con 0x5642cbdb5bb0 2024-03-24T00:40:54.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.588+0000 7fe13038c740 10 client.25757 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:54.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.588+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5642cbe32410 con 0x5642cbdb5bb0 2024-03-24T00:40:54.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.590+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe114005040 con 0x5642cbdb5bb0 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.590+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe120082070 con 0x5642cbdb5bb0 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.592+0000 7fe13038c740 10 client.25757 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.592+0000 7fe13038c740 10 client.25757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.592+0000 7fe13038c740 20 client.25757 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.592+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbe32d10 0x5642cbe35130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.593+0000 7fe13038c740 4 client.25757 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:40:54.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.593+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5642cbc9a3a0 con 0x5642cbe32d10 2024-03-24T00:40:54.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.594+0000 7fe126a00640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbe32d10 0x5642cbe35130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.598+0000 7fe126a00640 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbe32d10 0x5642cbe35130 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4afaff259d1f44d9 server_cookie=efc61ebe3103e501 in_seq=0 out_seq=0 2024-03-24T00:40:54.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.598+0000 7fe107e00640 10 client.25757 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:54.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.598+0000 7fe107e00640 1 -- 172.21.15.123:0/3198540786 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5642cbc9a3a0 con 0x5642cbe32d10 2024-03-24T00:40:54.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.599+0000 7fe107e00640 10 client.25757 handle_command_reply: tid=0 2024-03-24T00:40:54.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.599+0000 7fe13038c740 1 client.25757 shutdown 2024-03-24T00:40:54.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.600+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 msgr2=0x5642cbdc02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe1200049d0 tx=0x7fe1200058d0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 shutdown_connections 2024-03-24T00:40:54.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642cbdb5bb0 0x5642cbdc02f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5642cbe32d10 0x5642cbe35130 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5642cbdb0a40 0x5642cbdbd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 --2- 172.21.15.123:0/3198540786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5642cbd78200 0x5642cbdbcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 >> 172.21.15.123:0/3198540786 conn(0x5642cbdc82a0 msgr2=0x5642cbdb1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.601+0000 7fe107e00640 0 client.25757 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:54.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.602+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 shutdown_connections 2024-03-24T00:40:54.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.602+0000 7fe13038c740 1 -- 172.21.15.123:0/3198540786 wait complete. 2024-03-24T00:40:54.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.603+0000 7fe13038c740 20 client.25757 trim_cache size 0 max 16384 2024-03-24T00:40:54.605 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:54.605 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:40:54.605 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:40:54.605 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:54.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.606+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 msgr2=0x7fe10c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.606+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5642cbdb12a0 tx=0x7fe114005fd0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.606+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 msgr2=0x7fe12819ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.606+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe12000eb10 tx=0x7fe12000c2d0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 shutdown_connections 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1281072f0 0x7fe12819ee70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1281049c0 0x7fe128197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 --2- 172.21.15.123:0/3979381808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe128102090 0x7fe1281978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 >> 172.21.15.123:0/3979381808 conn(0x7fe1280fba20 msgr2=0x7fe1281029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 shutdown_connections 2024-03-24T00:40:54.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.608+0000 7fe127e00640 1 -- 172.21.15.123:0/3979381808 wait complete. 2024-03-24T00:40:54.640 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:54 smithi123 ceph-mon[31487]: pgmap v475: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 45 MiB/s wr, 6.78k op/s 2024-03-24T00:40:54.646 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:40:54.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:54 smithi177 ceph-mon[36034]: pgmap v475: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 45 MiB/s wr, 6.78k op/s 2024-03-24T00:40:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:54 smithi204 ceph-mon[36345]: pgmap v475: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 45 MiB/s wr, 6.78k op/s 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 -- 172.21.15.123:0/3739378280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f34101510 msgr2=0x7f2f34101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 --2- 172.21.15.123:0/3739378280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f34101510 0x7f2f34101930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009860 tx=0x7f2f2402bb70 comp rx=0 tx=0).stop 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 -- 172.21.15.123:0/3739378280 shutdown_connections 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 --2- 172.21.15.123:0/3739378280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f341030c0 0x7f2f34109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 --2- 172.21.15.123:0/3739378280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34102700 0x7f2f34102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 --2- 172.21.15.123:0/3739378280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f34101510 0x7f2f34101930 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 -- 172.21.15.123:0/3739378280 >> 172.21.15.123:0/3739378280 conn(0x7f2f340fca80 msgr2=0x7f2f340feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 -- 172.21.15.123:0/3739378280 shutdown_connections 2024-03-24T00:40:54.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.943+0000 7f2f38e00640 1 -- 172.21.15.123:0/3739378280 wait complete. 2024-03-24T00:40:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 Processor -- start 2024-03-24T00:40:54.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 -- start start 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 0x7f2f34110910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f341030c0 0x7f2f341add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34075660 con 0x7f2f34101510 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340754e0 con 0x7f2f34102700 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f38e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340757e0 con 0x7f2f341030c0 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.944+0000 7f2f32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47104/0 (socket says 172.21.15.123:47104) 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 -- 172.21.15.123:0/1721623063 learned_addr learned my addr 172.21.15.123:0/1721623063 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f33400640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 0x7f2f34110910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f33e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f341030c0 0x7f2f341add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 -- 172.21.15.123:0/1721623063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f341030c0 msgr2=0x7f2f341add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f341030c0 0x7f2f341add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 -- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 msgr2=0x7f2f34110910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 0x7f2f34110910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 -- 172.21.15.123:0/1721623063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f341ae290 con 0x7f2f34102700 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f33400640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 0x7f2f34110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f32a00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000e9f0 tx=0x7f2f2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.945+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000cd80 con 0x7f2f34102700 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.946+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f20010910 con 0x7f2f34102700 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.946+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f341ae580 con 0x7f2f34102700 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.946+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20018990 con 0x7f2f34102700 2024-03-24T00:40:54.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.946+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f341aeaa0 con 0x7f2f34102700 2024-03-24T00:40:54.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.947+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f34108830 con 0x7f2f34102700 2024-03-24T00:40:54.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.949+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2f20010ac0 con 0x7f2f34102700 2024-03-24T00:40:54.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.951+0000 7f2f2be00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:54.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.951+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2f200cede0 con 0x7f2f34102700 2024-03-24T00:40:54.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.951+0000 7f2f33400640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:54.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.952+0000 7f2f33400640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 0x7f2f180a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f240349e0 tx=0x7f2f2402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:54.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:54.955+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f2009b7e0 con 0x7f2f34102700 2024-03-24T00:40:55.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.209+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f340622a0 con 0x7f2f34102700 2024-03-24T00:40:55.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.210+0000 7f2f2be00640 1 -- 172.21.15.123:0/1721623063 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f2f2009f880 con 0x7f2f34102700 2024-03-24T00:40:55.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:40:55.211 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:40:55.211 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 msgr2=0x7f2f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 0x7f2f180a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f240349e0 tx=0x7f2f2402e040 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 msgr2=0x7f2f34110e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000e9f0 tx=0x7f2f2000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 shutdown_connections 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f341030c0 0x7f2f341add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f34102700 0x7f2f34110e50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 --2- 172.21.15.123:0/1721623063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f34101510 0x7f2f34110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 >> 172.21.15.123:0/1721623063 conn(0x7f2f340fca80 msgr2=0x7f2f34105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 shutdown_connections 2024-03-24T00:40:55.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.213+0000 7f2f38e00640 1 -- 172.21.15.123:0/1721623063 wait complete. 2024-03-24T00:40:55.230 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:40:55.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 -- 172.21.15.123:0/1822664855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 msgr2=0x7faebc1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 --2- 172.21.15.123:0/1822664855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1099e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009ca0 tx=0x7faeb402c100 comp rx=0 tx=0).stop 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 -- 172.21.15.123:0/1822664855 shutdown_connections 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 --2- 172.21.15.123:0/1822664855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1099e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 --2- 172.21.15.123:0/1822664855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc102790 0x7faebc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 --2- 172.21.15.123:0/1822664855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc1015a0 0x7faebc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.424+0000 7faec2400640 1 -- 172.21.15.123:0/1822664855 >> 172.21.15.123:0/1822664855 conn(0x7faebc0fcb10 msgr2=0x7faebc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.425+0000 7faec2400640 1 -- 172.21.15.123:0/1822664855 shutdown_connections 2024-03-24T00:40:55.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.425+0000 7faec2400640 1 -- 172.21.15.123:0/1822664855 wait complete. 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 Processor -- start 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 -- start start 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc1015a0 0x7faebc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d230 con 0x7faebc1015a0 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d0b0 con 0x7faebc102790 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec2400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d3b0 con 0x7faebc103150 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faebbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faebbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47134/0 (socket says 172.21.15.123:47134) 2024-03-24T00:40:55.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faebbe00640 1 -- 172.21.15.123:0/3755997332 learned_addr learned my addr 172.21.15.123:0/3755997332 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.426+0000 7faec1a00640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faec1000640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc1015a0 0x7faebc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 -- 172.21.15.123:0/3755997332 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 msgr2=0x7faebc1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 -- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc1015a0 msgr2=0x7faebc199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc1015a0 0x7faebc199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 -- 172.21.15.123:0/3755997332 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc1a1680 con 0x7faebc102790 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faec1a00640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faebbe00640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faea800e9f0 tx=0x7faea800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faeb9600640 1 -- 172.21.15.123:0/3755997332 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea800cd80 con 0x7faebc102790 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faeb9600640 1 -- 172.21.15.123:0/3755997332 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faea8010910 con 0x7faebc102790 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faeb9600640 1 -- 172.21.15.123:0/3755997332 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8018990 con 0x7faebc102790 2024-03-24T00:40:55.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc1a1970 con 0x7faebc102790 2024-03-24T00:40:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.427+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc1a1ec0 con 0x7faebc102790 2024-03-24T00:40:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.430+0000 7faeb9600640 1 -- 172.21.15.123:0/3755997332 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faea8010ac0 con 0x7faebc102790 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 Processor -- start 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 -- start start 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b335040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b33be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b2982c0 con 0x56259b335580 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b240c90 con 0x56259b3339f0 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec6621740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b294360 con 0x56259b334bc0 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47140/0 (socket says 172.21.15.123:47140) 2024-03-24T00:40:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faec1a00640 1 -- 172.21.15.123:0/4180374530 learned_addr learned my addr 172.21.15.123:0/4180374530 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.431+0000 7faebbe00640 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b33be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1000640 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b335040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 -- 172.21.15.123:0/4180374530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 msgr2=0x56259b335040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b335040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 -- 172.21.15.123:0/4180374530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 msgr2=0x56259b33be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b33be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 -- 172.21.15.123:0/4180374530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259b265080 con 0x56259b3339f0 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faeb9600640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faec1a00640 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faeb402c710 tx=0x7faeb40323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65766498e9b75e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.432+0000 7faeb9600640 1 -- 172.21.15.123:0/3755997332 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faea80ceeb0 con 0x7faebc102790 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faec1000640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faeae000640 1 -- 172.21.15.123:0/4180374530 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4043070 con 0x56259b3339f0 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faeae000640 1 -- 172.21.15.123:0/4180374530 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faeb4032c90 con 0x56259b3339f0 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faeae000640 1 -- 172.21.15.123:0/4180374530 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4039040 con 0x56259b3339f0 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faec6621740 1 -- 172.21.15.123:0/4180374530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 msgr2=0x56259b333e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faec1000640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 0x7faea40a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faebc1025f0 tx=0x7faeb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.433+0000 7faec6621740 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faeb402c710 tx=0x7faeb40323d0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 -- 172.21.15.123:0/4180374530 shutdown_connections 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b33be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b335040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 --2- 172.21.15.123:0/4180374530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b333e10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 -- 172.21.15.123:0/4180374530 >> 172.21.15.123:0/4180374530 conn(0x56259b32ede0 msgr2=0x56259b32f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 -- 172.21.15.123:0/4180374530 shutdown_connections 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 -- 172.21.15.123:0/4180374530 wait complete. 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 -- start start 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.434+0000 7faec6621740 1 Processor -- start 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b33e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b345180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b33c530 con 0x56259b335580 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b2540d0 con 0x56259b3339f0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b33c6b0 con 0x56259b334bc0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec6621740 10 client.0 _ll_register_callbacks cb 0x56259b258e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b345180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47144/0 (socket says 172.21.15.123:47144) 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 -- 172.21.15.123:0/210439679 learned_addr learned my addr 172.21.15.123:0/210439679 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faebbe00640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b33e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 -- 172.21.15.123:0/210439679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 msgr2=0x56259b33e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b33e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 -- 172.21.15.123:0/210439679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 msgr2=0x56259b345180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b345180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.435+0000 7faec1000640 1 -- 172.21.15.123:0/210439679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259afac3f0 con 0x56259b3339f0 2024-03-24T00:40:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faebbe00640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b33e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faec1000640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faeb0011c10 tx=0x7faeb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b678b34e85279a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faea2a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0011d00 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faeb4032c90 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faeb00108d0 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb0011ea0 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faec6621740 10 client.25775 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.436+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56259b0767d0 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faeb4032c90 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faeb0028cb0 con 0x56259b3339f0 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 10 client.25775 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 20 client.25775 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:55.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b34ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 4 client.25775 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:55.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56259b2a5800 con 0x56259b349620 2024-03-24T00:40:55.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.437+0000 7faebbe00640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b34ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.438+0000 7faea2a00640 10 client.25775 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.438+0000 7faebbe00640 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b34ba80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2de68ad2bdae8aba server_cookie=de8895cd3826eaa8 in_seq=0 out_seq=0 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.438+0000 7faea2a00640 1 -- 172.21.15.123:0/210439679 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56259b2a5800 con 0x56259b349620 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.438+0000 7faea2a00640 10 client.25775 handle_command_reply: tid=0 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 client.25775 shutdown 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 msgr2=0x56259b33dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faeb0011c10 tx=0x7faeb00046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 shutdown_connections 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b335580 0x56259b345180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b34ba80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b33e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faea2a00640 0 client.25775 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 --2- 172.21.15.123:0/210439679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b3339f0 0x56259b33dc00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 >> 172.21.15.123:0/210439679 conn(0x56259b32ede0 msgr2=0x56259b32e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 shutdown_connections 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 1 -- 172.21.15.123:0/210439679 wait complete. 2024-03-24T00:40:55.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.439+0000 7faec6621740 20 client.25775 trim_cache size 0 max 16384 2024-03-24T00:40:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.453+0000 7faec6621740 1 Processor -- start 2024-03-24T00:40:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.453+0000 7faec6621740 1 -- start start 2024-03-24T00:40:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b32d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b335580 0x56259b32de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1acfc0 con 0x56259b3339f0 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1af560 con 0x56259b335580 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec6621740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1ad5c0 con 0x56259b334bc0 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b335580 0x56259b32de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34932/0 (socket says 172.21.15.123:34932) 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faec1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b32d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 -- 172.21.15.123:0/1450384170 learned_addr learned my addr 172.21.15.123:0/1450384170 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 -- 172.21.15.123:0/1450384170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b335580 msgr2=0x56259b32de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b335580 0x56259b32de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 -- 172.21.15.123:0/1450384170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 msgr2=0x56259b32d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b32d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.454+0000 7faebbe00640 1 -- 172.21.15.123:0/1450384170 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259b33d020 con 0x56259b334bc0 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faebbe00640 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faea809b790 tx=0x7faea80ce040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1368359a6f8de3b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faead600640 1 -- 172.21.15.123:0/1450384170 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea809b160 con 0x56259b334bc0 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faead600640 1 -- 172.21.15.123:0/1450384170 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faea809bf00 con 0x56259b334bc0 2024-03-24T00:40:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faead600640 1 -- 172.21.15.123:0/1450384170 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8018b30 con 0x56259b334bc0 2024-03-24T00:40:55.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faec6621740 1 -- 172.21.15.123:0/1450384170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 msgr2=0x56259b345140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.455+0000 7faec6621740 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faea809b790 tx=0x7faea80ce040 comp rx=0 tx=0).stop 2024-03-24T00:40:55.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 -- 172.21.15.123:0/1450384170 shutdown_connections 2024-03-24T00:40:55.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b335580 0x56259b32de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b334bc0 0x56259b345140 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 --2- 172.21.15.123:0/1450384170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b32d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 -- 172.21.15.123:0/1450384170 >> 172.21.15.123:0/1450384170 conn(0x56259b32ede0 msgr2=0x56259b33baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 -- 172.21.15.123:0/1450384170 shutdown_connections 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 -- 172.21.15.123:0/1450384170 wait complete. 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 -- start start 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.456+0000 7faec6621740 1 Processor -- start 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b340bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b334bc0 0x56259b341110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1adbe0 con 0x56259b3339f0 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1aef00 con 0x56259b334bc0 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1ae240 con 0x56259b335580 2024-03-24T00:40:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec6621740 10 client.0 _ll_register_callbacks cb 0x7faeb0009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b340bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34942/0 (socket says 172.21.15.123:34942) 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 -- 172.21.15.123:0/3844400724 learned_addr learned my addr 172.21.15.123:0/3844400724 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faebbe00640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b334bc0 0x56259b341110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 -- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b334bc0 msgr2=0x56259b341110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b334bc0 0x56259b341110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 -- 172.21.15.123:0/3844400724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 msgr2=0x56259b340bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b340bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1a00640 1 -- 172.21.15.123:0/3844400724 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259b3446d0 con 0x56259b335580 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.457+0000 7faec1000640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b340bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faec1a00640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009bf0 tx=0x7faeb4009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dab1431fecac2b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faea2a00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:55.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4055060 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faeb4003970 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faea809bf00 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb4039050 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faec6621740 10 client.25777 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.458+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56259b3a5260 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faea809bf00 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faeb40040d0 con 0x56259b335580 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 10 client.25777 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 10 client.25777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 20 client.25777 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b3a7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 4 client.25777 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56259b29fb50 con 0x56259b349620 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec1000640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b3a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faec1000640 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b3a7880 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db1517891f33a8f server_cookie=f24f0d48fea9fcc9 in_seq=0 out_seq=0 2024-03-24T00:40:55.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.459+0000 7faea2a00640 10 client.25777 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faea2a00640 1 -- 172.21.15.123:0/3844400724 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56259b29fb50 con 0x56259b349620 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faea2a00640 10 client.25777 handle_command_reply: tid=0 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 client.25777 shutdown 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 msgr2=0x56259b3440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009bf0 tx=0x7faeb4009880 comp rx=0 tx=0).stop 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 shutdown_connections 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56259b349620 0x56259b3a7880 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56259b335580 0x56259b3440e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56259b334bc0 0x56259b341110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 --2- 172.21.15.123:0/3844400724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56259b3339f0 0x56259b340bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faea2a00640 0 client.25777 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 >> 172.21.15.123:0/3844400724 conn(0x56259b32ede0 msgr2=0x56259b332040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 shutdown_connections 2024-03-24T00:40:55.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 1 -- 172.21.15.123:0/3844400724 wait complete. 2024-03-24T00:40:55.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.460+0000 7faec6621740 20 client.25777 trim_cache size 0 max 16384 2024-03-24T00:40:55.465 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 msgr2=0x7faea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 0x7faea40a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faebc1025f0 tx=0x7faeb0005f90 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 msgr2=0x7faebc199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faea800e9f0 tx=0x7faea800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 shutdown_connections 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faebc103150 0x7faebc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc102790 0x7faebc199fa0 secure :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faea800e9f0 tx=0x7faea800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 --2- 172.21.15.123:0/3755997332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc1015a0 0x7faebc199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 >> 172.21.15.123:0/3755997332 conn(0x7faebc0fcb10 msgr2=0x7faebc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 shutdown_connections 2024-03-24T00:40:55.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:55.461+0000 7faec2400640 1 -- 172.21.15.123:0/3755997332 wait complete. 2024-03-24T00:40:55.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1721623063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:55.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1721623063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:55.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1721623063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:40:56.478 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:40:56.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:56.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 -- 172.21.15.123:0/2498113521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 msgr2=0x7fe114102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 --2- 172.21.15.123:0/2498113521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114102c10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009900 tx=0x7fe0fc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 -- 172.21.15.123:0/2498113521 shutdown_connections 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 --2- 172.21.15.123:0/2498113521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 0x7fe1141099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 --2- 172.21.15.123:0/2498113521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114102c10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 --2- 172.21.15.123:0/2498113521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 -- 172.21.15.123:0/2498113521 >> 172.21.15.123:0/2498113521 conn(0x7fe1140fcb10 msgr2=0x7fe1140fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 -- 172.21.15.123:0/2498113521 shutdown_connections 2024-03-24T00:40:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.675+0000 7fe113200640 1 -- 172.21.15.123:0/2498113521 wait complete. 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 Processor -- start 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 -- start start 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 0x7fe11419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11410d270 con 0x7fe114103150 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11410d0f0 con 0x7fe114102790 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.676+0000 7fe113200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11410d3f0 con 0x7fe1141015a0 2024-03-24T00:40:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49926/0 (socket says 172.21.15.123:49926) 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 -- 172.21.15.123:0/4008097116 learned_addr learned my addr 172.21.15.123:0/4008097116 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe112800640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 0x7fe11419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111400640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 -- 172.21.15.123:0/4008097116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 msgr2=0x7fe114197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 -- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 msgr2=0x7fe11419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 0x7fe11419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 -- 172.21.15.123:0/4008097116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11419f4e0 con 0x7fe1141015a0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.677+0000 7fe111e00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe10400eb70 tx=0x7fe10400ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.678+0000 7fe10aa00640 1 -- 172.21.15.123:0/4008097116 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104003970 con 0x7fe1141015a0 2024-03-24T00:40:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.678+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11419f7d0 con 0x7fe1141015a0 2024-03-24T00:40:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.678+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11419fcf0 con 0x7fe1141015a0 2024-03-24T00:40:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.678+0000 7fe10aa00640 1 -- 172.21.15.123:0/4008097116 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe104004020 con 0x7fe1141015a0 2024-03-24T00:40:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.678+0000 7fe10aa00640 1 -- 172.21.15.123:0/4008097116 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104010670 con 0x7fe1141015a0 2024-03-24T00:40:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe10aa00640 1 -- 172.21.15.123:0/4008097116 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe104010950 con 0x7fe1141015a0 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 Processor -- start 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 -- start start 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8dca0 0x5626c9e927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e952a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9df2eb0 con 0x5626c9e8dca0 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9d9ba60 con 0x5626c9e92e10 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.681+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9deef50 con 0x5626c9e8d2b0 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49932/0 (socket says 172.21.15.123:49932) 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 -- 172.21.15.123:0/1137319652 learned_addr learned my addr 172.21.15.123:0/1137319652 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe111e00640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8dca0 0x5626c9e927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe111400640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 -- 172.21.15.123:0/1137319652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 msgr2=0x5626c9e952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 -- 172.21.15.123:0/1137319652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8dca0 msgr2=0x5626c9e927a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8dca0 0x5626c9e927a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 -- 172.21.15.123:0/1137319652 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c9bcf7e0 con 0x5626c9e8d2b0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe111400640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe10aa00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 0x7fe0f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.682+0000 7fe112800640 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009c90 tx=0x7fe10c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcebbe46be862480 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe10aa00640 1 -- 172.21.15.123:0/4008097116 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe1040cf050 con 0x7fe1141015a0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe111e00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 0x7fe0f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe103400640 1 -- 172.21.15.123:0/1137319652 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c004230 con 0x5626c9e8d2b0 2024-03-24T00:40:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe103400640 1 -- 172.21.15.123:0/1137319652 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe10c009e40 con 0x5626c9e8d2b0 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe103400640 1 -- 172.21.15.123:0/1137319652 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0197e0 con 0x5626c9e8d2b0 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe11bbbd740 1 -- 172.21.15.123:0/1137319652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 msgr2=0x5626c9e8d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe111e00640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 0x7fe0f80a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe1040cced0 tx=0x7fe1040a1050 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.683+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009c90 tx=0x7fe10c011810 comp rx=0 tx=0).stop 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 -- 172.21.15.123:0/1137319652 shutdown_connections 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8dca0 0x5626c9e927a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/1137319652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8d6d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 -- 172.21.15.123:0/1137319652 >> 172.21.15.123:0/1137319652 conn(0x5626c9e88820 msgr2=0x5626c9e88c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 -- 172.21.15.123:0/1137319652 shutdown_connections 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 -- 172.21.15.123:0/1137319652 wait complete. 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 -- start start 2024-03-24T00:40:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.684+0000 7fe11bbbd740 1 Processor -- start 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 0x5626c9e97490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e92e10 0x5626c9e9ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9e95dc0 con 0x5626c9e92e10 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9daeeb0 con 0x5626c9e8d2b0 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c9e95f40 con 0x5626c9e8dca0 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe11bbbd740 10 client.0 _ll_register_callbacks cb 0x5626c9da3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe112800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 0x5626c9e97490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49944/0 (socket says 172.21.15.123:49944) 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe112800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 0x5626c9e97490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44946/0 (socket says 172.21.15.123:44946) 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 -- 172.21.15.123:0/3815100312 learned_addr learned my addr 172.21.15.123:0/3815100312 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe112800640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e92e10 0x5626c9e9ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 -- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 msgr2=0x5626c9e97490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 0x5626c9e97490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 -- 172.21.15.123:0/3815100312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e92e10 msgr2=0x5626c9e9ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e92e10 0x5626c9e9ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.685+0000 7fe111400640 1 -- 172.21.15.123:0/3815100312 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c9c2ff70 con 0x5626c9e8dca0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe111400640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc005ab0 tx=0x7fe0fc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3306a8ef28973dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe0f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc005d50 con 0x5626c9e8dca0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe0fc0048d0 con 0x5626c9e8dca0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626c9e003f0 con 0x5626c9e8dca0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0fc0044e0 con 0x5626c9e8dca0 2024-03-24T00:40:56.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe11bbbd740 10 client.25789 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.686+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5626c9ac7170 con 0x5626c9e8dca0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5626c9e003f0 con 0x5626c9e8dca0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe0fc0035f0 con 0x5626c9e8dca0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 10 client.25789 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 10 client.25789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 20 client.25789 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9ea3370 0x5626c9ea5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 4 client.25789 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626c9dfa740 con 0x5626c9ea3370 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.687+0000 7fe112800640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9ea3370 0x5626c9ea5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.688+0000 7fe112800640 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9ea3370 0x5626c9ea5790 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd28e3c2ba61c0a7 server_cookie=ec2783b97bd7c47e in_seq=0 out_seq=0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.688+0000 7fe0f7e00640 10 client.25789 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.688+0000 7fe0f7e00640 1 -- 172.21.15.123:0/3815100312 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626c9dfa740 con 0x5626c9ea3370 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.688+0000 7fe0f7e00640 10 client.25789 handle_command_reply: tid=0 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.688+0000 7fe11bbbd740 1 client.25789 shutdown 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 msgr2=0x5626c9e979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc005ab0 tx=0x7fe0fc0042f0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 shutdown_connections 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e92e10 0x5626c9e9ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9ea3370 0x5626c9ea5790 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e979d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/3815100312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8d2b0 0x5626c9e97490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe0f7e00640 0 client.25789 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 >> 172.21.15.123:0/3815100312 conn(0x5626c9e88820 msgr2=0x5626c9e87a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 shutdown_connections 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 1 -- 172.21.15.123:0/3815100312 wait complete. 2024-03-24T00:40:56.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.689+0000 7fe11bbbd740 20 client.25789 trim_cache size 0 max 16384 2024-03-24T00:40:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:56 smithi177 ceph-mon[36034]: pgmap v476: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 49 MiB/s wr, 7.30k op/s 2024-03-24T00:40:56.722 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:56 smithi204 ceph-mon[36345]: pgmap v476: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 49 MiB/s wr, 7.30k op/s 2024-03-24T00:40:56.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 Processor -- start 2024-03-24T00:40:56.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 -- start start 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e92e10 0x5626c9e8ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000c3c0 con 0x5626c9e8d2b0 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.721+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000e980 con 0x5626c9e8dca0 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.722+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000c9c0 con 0x5626c9e92e10 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.722+0000 7fe111400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.722+0000 7fe111400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44962/0 (socket says 172.21.15.123:44962) 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.722+0000 7fe111400640 1 -- 172.21.15.123:0/404405165 learned_addr learned my addr 172.21.15.123:0/404405165 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:56.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.722+0000 7fe112800640 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe111400640 1 -- 172.21.15.123:0/404405165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e92e10 msgr2=0x5626c9e8ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe111400640 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e92e10 0x5626c9e8ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe111400640 1 -- 172.21.15.123:0/404405165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 msgr2=0x5626c9e8e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe111400640 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe111400640 1 -- 172.21.15.123:0/404405165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c9e88650 con 0x5626c9e8dca0 2024-03-24T00:40:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.723+0000 7fe112800640 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:56.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.725+0000 7fe111400640 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9cc0d805fe8188a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.726+0000 7fe102a00640 1 -- 172.21.15.123:0/404405165 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc059070 con 0x5626c9e8dca0 2024-03-24T00:40:56.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.726+0000 7fe102a00640 1 -- 172.21.15.123:0/404405165 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0fc009780 con 0x5626c9e8dca0 2024-03-24T00:40:56.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.726+0000 7fe102a00640 1 -- 172.21.15.123:0/404405165 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc000e60 con 0x5626c9e8dca0 2024-03-24T00:40:56.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.728+0000 7fe11bbbd740 1 -- 172.21.15.123:0/404405165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 msgr2=0x5626c9e9e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.728+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc005fd0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.729+0000 7fe11bbbd740 1 -- 172.21.15.123:0/404405165 shutdown_connections 2024-03-24T00:40:56.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.729+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e92e10 0x5626c9e8ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.729+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e8dca0 0x5626c9e9e9d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.729+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/404405165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.729+0000 7fe11bbbd740 1 -- 172.21.15.123:0/404405165 >> 172.21.15.123:0/404405165 conn(0x5626c9e88820 msgr2=0x5626c9e96bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.730+0000 7fe11bbbd740 1 -- 172.21.15.123:0/404405165 shutdown_connections 2024-03-24T00:40:56.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.730+0000 7fe11bbbd740 1 -- 172.21.15.123:0/404405165 wait complete. 2024-03-24T00:40:56.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.731+0000 7fe11bbbd740 1 -- start start 2024-03-24T00:40:56.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.732+0000 7fe11bbbd740 1 Processor -- start 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e9abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e9e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000d000 con 0x5626c9e8d2b0 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000e320 con 0x5626c9e92e10 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000d660 con 0x5626c9e8dca0 2024-03-24T00:40:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.733+0000 7fe11bbbd740 10 client.0 _ll_register_callbacks cb 0x7fe1040093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:56.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.734+0000 7fe111400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.734+0000 7fe111400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49952/0 (socket says 172.21.15.123:49952) 2024-03-24T00:40:56.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.734+0000 7fe111400640 1 -- 172.21.15.123:0/200835135 learned_addr learned my addr 172.21.15.123:0/200835135 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:56.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.735+0000 7fe111400640 1 -- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 msgr2=0x5626c9e9e0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:40:56.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.735+0000 7fe111400640 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e9e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.735+0000 7fe111400640 1 -- 172.21.15.123:0/200835135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 msgr2=0x5626c9e9abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:56.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.735+0000 7fe111400640 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e9abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.735+0000 7fe111400640 1 -- 172.21.15.123:0/200835135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c9e89290 con 0x5626c9e8dca0 2024-03-24T00:40:56.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.736+0000 7fe111400640 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc003b30 tx=0x7fe0fc003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a356d184a472a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:56.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.736+0000 7fe0f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:56.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.737+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc059040 con 0x5626c9e8dca0 2024-03-24T00:40:56.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.737+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe0fc004eb0 con 0x5626c9e8dca0 2024-03-24T00:40:56.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.738+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0fc009780 con 0x5626c9e8dca0 2024-03-24T00:40:56.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.738+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0fc004ce0 con 0x5626c9e8dca0 2024-03-24T00:40:56.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.738+0000 7fe11bbbd740 10 client.25792 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:56.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.738+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5626c9f0fee0 con 0x5626c9e8dca0 2024-03-24T00:40:56.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.740+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0fc009780 con 0x5626c9e8dca0 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.741+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe0fc04a8c0 con 0x5626c9e8dca0 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 10 client.25792 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 20 client.25792 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9f10820 0x5626c9f12cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 4 client.25792 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:40:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.742+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5626c9d78a40 con 0x5626c9f10820 2024-03-24T00:40:56.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.746+0000 7fe112800640 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9f10820 0x5626c9f12cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:56.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.749+0000 7fe112800640 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9f10820 0x5626c9f12cc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef82c53da6bf4417 server_cookie=dc9e27a634051c6b in_seq=0 out_seq=0 2024-03-24T00:40:56.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.750+0000 7fe0f7e00640 10 client.25792 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:56.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.765+0000 7fe0f7e00640 1 -- 172.21.15.123:0/200835135 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5626c9d78a40 con 0x5626c9f10820 2024-03-24T00:40:56.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.765+0000 7fe0f7e00640 10 client.25792 handle_command_reply: tid=0 2024-03-24T00:40:56.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.765+0000 7fe11bbbd740 1 client.25792 shutdown 2024-03-24T00:40:56.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.766+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 msgr2=0x5626c9e9b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.767+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc003b30 tx=0x7fe0fc003680 comp rx=0 tx=0).stop 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 shutdown_connections 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626c9e92e10 0x5626c9e9e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626c9f10820 0x5626c9f12cc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626c9e8dca0 0x5626c9e9b100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 --2- 172.21.15.123:0/200835135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626c9e8d2b0 0x5626c9e9abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 >> 172.21.15.123:0/200835135 conn(0x5626c9e88820 msgr2=0x5626c9e871f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.770+0000 7fe0f7e00640 0 client.25792 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:56.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.771+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 shutdown_connections 2024-03-24T00:40:56.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.771+0000 7fe11bbbd740 1 -- 172.21.15.123:0/200835135 wait complete. 2024-03-24T00:40:56.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.771+0000 7fe11bbbd740 20 client.25792 trim_cache size 0 max 16384 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "fe59a8f1-2961-4333-bf81-9f751fdb5f78", 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:56.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.774+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 msgr2=0x7fe0f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.775+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 0x7fe0f80a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe1040cced0 tx=0x7fe1040a1050 comp rx=0 tx=0).stop 2024-03-24T00:40:56.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.775+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 msgr2=0x7fe1141978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:56.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.775+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe10400eb70 tx=0x7fe10400ef70 comp rx=0 tx=0).stop 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 shutdown_connections 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe0f80a2880 0x7fe0f80a4d40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe114103150 0x7fe11419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114102790 0x7fe114197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 --2- 172.21.15.123:0/4008097116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1141015a0 0x7fe1141978c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:56.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.776+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 >> 172.21.15.123:0/4008097116 conn(0x7fe1140fcb10 msgr2=0x7fe114105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:56.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.779+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 shutdown_connections 2024-03-24T00:40:56.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:56.779+0000 7fe113200640 1 -- 172.21.15.123:0/4008097116 wait complete. 2024-03-24T00:40:56.819 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:56 smithi123 ceph-mon[31487]: pgmap v476: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 49 MiB/s wr, 7.30k op/s 2024-03-24T00:40:56.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:40:56.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.152+0000 7f1c7a200640 1 -- 172.21.15.123:0/72671113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 msgr2=0x7f1c74104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.152+0000 7f1c7a200640 1 --2- 172.21.15.123:0/72671113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c74104f50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009900 tx=0x7f1c6002bc50 comp rx=0 tx=0).stop 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 -- 172.21.15.123:0/72671113 shutdown_connections 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 --2- 172.21.15.123:0/72671113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c74105490 0x7f1c7410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 --2- 172.21.15.123:0/72671113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c74104f50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 --2- 172.21.15.123:0/72671113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c741038e0 0x7f1c74103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 -- 172.21.15.123:0/72671113 >> 172.21.15.123:0/72671113 conn(0x7f1c740fee50 msgr2=0x7f1c74101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 -- 172.21.15.123:0/72671113 shutdown_connections 2024-03-24T00:40:57.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.153+0000 7f1c7a200640 1 -- 172.21.15.123:0/72671113 wait complete. 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 Processor -- start 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 -- start start 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c7419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c74105490 0x7f1c741a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7410f590 con 0x7f1c74105490 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7410f410 con 0x7f1c741038e0 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c7a200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7410f710 con 0x7f1c74104ad0 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.154+0000 7f1c73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44972/0 (socket says 172.21.15.123:44972) 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c73400640 1 -- 172.21.15.123:0/4114737916 learned_addr learned my addr 172.21.15.123:0/4114737916 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:57.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c7419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c73e00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c74105490 0x7f1c741a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 -- 172.21.15.123:0/4114737916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 msgr2=0x7f1c74199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 -- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c74105490 msgr2=0x7f1c741a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c74105490 0x7f1c741a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c72a00640 1 -- 172.21.15.123:0/4114737916 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74075b60 con 0x7f1c74104ad0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.155+0000 7f1c73400640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c72a00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c7419a110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c60036040 tx=0x7f1c6002c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c6be00640 1 -- 172.21.15.123:0/4114737916 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60041070 con 0x7f1c74104ad0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c6be00640 1 -- 172.21.15.123:0/4114737916 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c60004450 con 0x7f1c74104ad0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c6be00640 1 -- 172.21.15.123:0/4114737916 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60034440 con 0x7f1c74104ad0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c74075df0 con 0x7f1c74104ad0 2024-03-24T00:40:57.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.156+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c74076200 con 0x7f1c74104ad0 2024-03-24T00:40:57.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.159+0000 7f1c6be00640 1 -- 172.21.15.123:0/4114737916 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1c60044070 con 0x7f1c74104ad0 2024-03-24T00:40:57.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.160+0000 7f1c7cfac740 1 Processor -- start 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.160+0000 7f1c7cfac740 1 -- start start 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.160+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3cda2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3cdf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3c3f250 con 0x557ab3cd9e80 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3be7a30 con 0x557ab3cdf9e0 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3c3b2f0 con 0x557ab3cda870 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c72a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c6be00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 0x7f1c580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c73400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3cdf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c72a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44974/0 (socket says 172.21.15.123:44974) 2024-03-24T00:40:57.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c72a00640 1 -- 172.21.15.123:0/653389884 learned_addr learned my addr 172.21.15.123:0/653389884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:57.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.161+0000 7f1c6be00640 1 -- 172.21.15.123:0/4114737916 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1c600f04d0 con 0x7f1c74104ad0 2024-03-24T00:40:57.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 -- 172.21.15.123:0/653389884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 msgr2=0x557ab3cdf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3cdf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 -- 172.21.15.123:0/653389884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 msgr2=0x557ab3cda2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3cda2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 -- 172.21.15.123:0/653389884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ab3bd2690 con 0x557ab3cdf9e0 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c73e00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 0x7f1c580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c72a00640 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1c600b9900 tx=0x7f1c6002c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb52902ae5a7815c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c5fe00640 1 -- 172.21.15.123:0/653389884 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6004b070 con 0x557ab3cdf9e0 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c5fe00640 1 -- 172.21.15.123:0/653389884 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c600c0fd0 con 0x557ab3cdf9e0 2024-03-24T00:40:57.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.162+0000 7f1c5fe00640 1 -- 172.21.15.123:0/653389884 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60003680 con 0x557ab3cdf9e0 2024-03-24T00:40:57.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.163+0000 7f1c7cfac740 1 -- 172.21.15.123:0/653389884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 msgr2=0x557ab3ce1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.163+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1c600b9900 tx=0x7f1c6002c8c0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.163+0000 7f1c73e00640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 0x7f1c580a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x557ab3cdb0d0 tx=0x7f1c6c009840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 -- 172.21.15.123:0/653389884 shutdown_connections 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce1e70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3cdf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/653389884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3cda2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 -- 172.21.15.123:0/653389884 >> 172.21.15.123:0/653389884 conn(0x557ab3ca2200 msgr2=0x557ab3cd57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 -- 172.21.15.123:0/653389884 shutdown_connections 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 -- 172.21.15.123:0/653389884 wait complete. 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.164+0000 7f1c7cfac740 1 -- start start 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.165+0000 7f1c7cfac740 1 Processor -- start 2024-03-24T00:40:57.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.165+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3ce3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.165+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.165+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ceb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3be7a30 con 0x557ab3cd9e80 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3bfae80 con 0x557ab3cdf9e0 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab3ce2750 con 0x557ab3cda870 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49994/0 (socket says 172.21.15.123:49994) 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 -- 172.21.15.123:0/39400935 learned_addr learned my addr 172.21.15.123:0/39400935 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c7cfac740 10 client.0 _ll_register_callbacks cb 0x557ab3c0d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c73e00640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ceb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 -- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 msgr2=0x557ab3ceb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ceb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c73400640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3ce3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 -- 172.21.15.123:0/39400935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 msgr2=0x557ab3ce3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3ce3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 -- 172.21.15.123:0/39400935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ab3a1d7d0 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c72a00640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1c600b9390 tx=0x7f1c600035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3790ecde391b101d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6004b070 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.166+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1c600bd190 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ab3c4c790 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c73400640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3ce3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c600bcf30 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c7cfac740 10 client.25804 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557ab39533f0 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557ab3c4c790 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.167+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1c600bc380 con 0x557ab3cda870 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 10 client.25804 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 10 client.25804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 20 client.25804 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3cef5a0 0x557ab3cf1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 4 client.25804 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ab3c46ae0 con 0x557ab3cef5a0 2024-03-24T00:40:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c73400640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3cef5a0 0x557ab3cf1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c73400640 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3cef5a0 0x557ab3cf1990 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd60bab5bd83ea0e server_cookie=17028c27f47554d1 in_seq=0 out_seq=0 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.168+0000 7f1c5cc00640 10 client.25804 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.169+0000 7f1c5cc00640 1 -- 172.21.15.123:0/39400935 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557ab3c46ae0 con 0x557ab3cef5a0 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.169+0000 7f1c5cc00640 10 client.25804 handle_command_reply: tid=0 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.169+0000 7f1c7cfac740 1 client.25804 shutdown 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.169+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 msgr2=0x557ab3ce3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.169+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1c600b9390 tx=0x7f1c600035f0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 shutdown_connections 2024-03-24T00:40:57.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ceb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3cef5a0 0x557ab3cf1990 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ce3fd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/39400935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cd9e80 0x557ab3ce3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 >> 172.21.15.123:0/39400935 conn(0x557ab3ca2200 msgr2=0x557ab3cd4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c5cc00640 0 client.25804 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 shutdown_connections 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 1 -- 172.21.15.123:0/39400935 wait complete. 2024-03-24T00:40:57.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.170+0000 7f1c7cfac740 20 client.25804 trim_cache size 0 max 16384 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.187+0000 7f1c7cfac740 1 Processor -- start 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.187+0000 7f1c7cfac740 1 -- start start 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ceafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cdf9e0 0x557ab3cf6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741ad1c0 con 0x557ab3cdf9e0 2024-03-24T00:40:57.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741af780 con 0x557ab3cd9e80 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741ad7c0 con 0x557ab3cda870 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44998/0 (socket says 172.21.15.123:44998) 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c73400640 1 -- 172.21.15.123:0/2974292053 learned_addr learned my addr 172.21.15.123:0/2974292053 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ceafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c73e00640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cdf9e0 0x557ab3cf6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 -- 172.21.15.123:0/2974292053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 msgr2=0x557ab3cdeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 -- 172.21.15.123:0/2974292053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cdf9e0 msgr2=0x557ab3cf6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cdf9e0 0x557ab3cf6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.188+0000 7f1c72a00640 1 -- 172.21.15.123:0/2974292053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ab39130f0 con 0x557ab3cda870 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.189+0000 7f1c73400640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.189+0000 7f1c72a00640 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ceafd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002c7d0 tx=0x7f1c60003a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e570f2cd1f0d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.189+0000 7f1c5f400640 1 -- 172.21.15.123:0/2974292053 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6004b070 con 0x557ab3cda870 2024-03-24T00:40:57.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.189+0000 7f1c5f400640 1 -- 172.21.15.123:0/2974292053 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c60096020 con 0x557ab3cda870 2024-03-24T00:40:57.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.189+0000 7f1c5f400640 1 -- 172.21.15.123:0/2974292053 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c600c5190 con 0x557ab3cda870 2024-03-24T00:40:57.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- 172.21.15.123:0/2974292053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 msgr2=0x557ab3ceafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ceafd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002c7d0 tx=0x7f1c60003a20 comp rx=0 tx=0).stop 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- 172.21.15.123:0/2974292053 shutdown_connections 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cdf9e0 0x557ab3cf6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cda870 0x557ab3ceafd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/2974292053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cd9e80 0x557ab3cdeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- 172.21.15.123:0/2974292053 >> 172.21.15.123:0/2974292053 conn(0x557ab3ca2200 msgr2=0x557ab3cdbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- 172.21.15.123:0/2974292053 shutdown_connections 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- 172.21.15.123:0/2974292053 wait complete. 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.190+0000 7f1c7cfac740 1 -- start start 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 Processor -- start 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 0x557ab3ce8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 0x557ab3ce8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741ade00 con 0x557ab3cda870 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741af120 con 0x557ab3cdf9e0 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741ae460 con 0x557ab3cd9e80 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.191+0000 7f1c7cfac740 10 client.0 _ll_register_callbacks cb 0x7f1c64009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 0x557ab3ce8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45002/0 (socket says 172.21.15.123:45002) 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73e00640 1 -- 172.21.15.123:0/1497941369 learned_addr learned my addr 172.21.15.123:0/1497941369 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c72a00640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 0x557ab3ce8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 -- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 msgr2=0x557ab3ce9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 -- 172.21.15.123:0/1497941369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 msgr2=0x557ab3ce8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 0x557ab3ce8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73400640 1 -- 172.21.15.123:0/1497941369 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ab3cea0d0 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c73e00640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.192+0000 7f1c72a00640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 0x557ab3ce8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c73400640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 0x557ab3ce8590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c64000c00 tx=0x7f1c6401fce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dcbcdb0c72c1cea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64004250 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1c640043f0 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c60096020 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c64012e70 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c7cfac740 10 client.25810 fetch_fsmap learned FSMap version 19 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557ab3d4ae00 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.193+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c60096020 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1c64026020 con 0x557ab3cd9e80 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 10 client.25810 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 10 client.25810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 20 client.25810 populate_metadata read hostname 'smithi123' 2024-03-24T00:40:57.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3d4b5a0 0x557ab3d4da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:57.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 4 client.25810 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:40:57.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557ab3bc43a0 con 0x557ab3d4b5a0 2024-03-24T00:40:57.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c72a00640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3d4b5a0 0x557ab3d4da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c72a00640 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3d4b5a0 0x557ab3d4da00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c1b0ee298795bf server_cookie=e9aa7222c3b0674a in_seq=0 out_seq=0 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.194+0000 7f1c5cc00640 10 client.25810 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c5cc00640 1 -- 172.21.15.123:0/1497941369 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x557ab3bc43a0 con 0x557ab3d4b5a0 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c5cc00640 10 client.25810 handle_command_reply: tid=0 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 client.25810 shutdown 2024-03-24T00:40:57.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 msgr2=0x557ab3ce8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 0x557ab3ce8590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c64000c00 tx=0x7f1c6401fce0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 shutdown_connections 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ab3cdf9e0 0x557ab3ce9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ab3cda870 0x557ab3ce8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557ab3d4b5a0 0x557ab3d4da00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 --2- 172.21.15.123:0/1497941369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557ab3cd9e80 0x557ab3ce8590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c5cc00640 0 client.25810 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 >> 172.21.15.123:0/1497941369 conn(0x557ab3ca2200 msgr2=0x557ab3cdbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 shutdown_connections 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 1 -- 172.21.15.123:0/1497941369 wait complete. 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.195+0000 7f1c7cfac740 20 client.25810 trim_cache size 0 max 16384 2024-03-24T00:40:57.199 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (79 inodes in the stack)", 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "fe59a8f1-2961-4333-bf81-9f751fdb5f78": { 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "fe59a8f1-2961-4333-bf81-9f751fdb5f78", 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:40:57.200 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 msgr2=0x7f1c580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 0x7f1c580a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x557ab3cdb0d0 tx=0x7f1c6c009840 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 msgr2=0x7f1c7419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c7419a110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c60036040 tx=0x7f1c6002c6e0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 shutdown_connections 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1c580a2880 0x7f1c580a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c74105490 0x7f1c741a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c74104ad0 0x7f1c7419a110 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 --2- 172.21.15.123:0/4114737916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c741038e0 0x7f1c74199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:57.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 >> 172.21.15.123:0/4114737916 conn(0x7f1c740fee50 msgr2=0x7f1c74107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:57.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 shutdown_connections 2024-03-24T00:40:57.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:40:57.196+0000 7f1c7a200640 1 -- 172.21.15.123:0/4114737916 wait complete. 2024-03-24T00:40:57.213 INFO:tasks.cephfs.filesystem:scrub status for tag:fe59a8f1-2961-4333-bf81-9f751fdb5f78 - {'path': '/', 'tag': 'fe59a8f1-2961-4333-bf81-9f751fdb5f78', 'options': 'recursive,force'} 2024-03-24T00:40:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:57 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:40:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:57 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:57 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:40:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:40:57 smithi123 ceph-mon[31487]: pgmap v477: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-24T00:40:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:57 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:40:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:57 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:57.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:57 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:40:57.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:40:57 smithi177 ceph-mon[36034]: pgmap v477: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-24T00:40:57.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:57 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:40:57.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:57 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:40:57.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:57 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:40:57.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:40:57 smithi204 ceph-mon[36345]: pgmap v477: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-24T00:41:00.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:00 smithi204 ceph-mon[36345]: pgmap v478: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 49 MiB/s wr, 6.77k op/s 2024-03-24T00:41:00.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:00 smithi123 ceph-mon[31487]: pgmap v478: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 49 MiB/s wr, 6.77k op/s 2024-03-24T00:41:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:00 smithi177 ceph-mon[36034]: pgmap v478: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 49 MiB/s wr, 6.77k op/s 2024-03-24T00:41:02.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:02 smithi204 ceph-mon[36345]: pgmap v479: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 55 MiB/s wr, 7.18k op/s 2024-03-24T00:41:02.608 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:02 smithi123 ceph-mon[31487]: pgmap v479: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 55 MiB/s wr, 7.18k op/s 2024-03-24T00:41:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:02 smithi177 ceph-mon[36034]: pgmap v479: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 55 MiB/s wr, 7.18k op/s 2024-03-24T00:41:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:04 smithi204 ceph-mon[36345]: pgmap v480: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 53 MiB/s wr, 6.42k op/s 2024-03-24T00:41:04.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:04 smithi123 ceph-mon[31487]: pgmap v480: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 53 MiB/s wr, 6.42k op/s 2024-03-24T00:41:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:04 smithi177 ceph-mon[36034]: pgmap v480: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 53 MiB/s wr, 6.42k op/s 2024-03-24T00:41:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:06 smithi204 ceph-mon[36345]: pgmap v481: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 55 MiB/s wr, 6.45k op/s 2024-03-24T00:41:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:06 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:06.610 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:06 smithi123 ceph-mon[31487]: pgmap v481: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 55 MiB/s wr, 6.45k op/s 2024-03-24T00:41:06.610 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:06 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:06 smithi177 ceph-mon[36034]: pgmap v481: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 55 MiB/s wr, 6.45k op/s 2024-03-24T00:41:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:06 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:07 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:07 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:41:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:07.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:07 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:41:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:07 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:07 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:41:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:07.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:07.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:07 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:41:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 -- 172.21.15.123:0/1691214802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20102700 msgr2=0x7f7d20102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1691214802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20102700 0x7f7d20102b80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009900 tx=0x7f7d0c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:41:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 -- 172.21.15.123:0/1691214802 shutdown_connections 2024-03-24T00:41:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1691214802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d201030c0 0x7f7d20109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1691214802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20102700 0x7f7d20102b80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1691214802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d20101510 0x7f7d20101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 -- 172.21.15.123:0/1691214802 >> 172.21.15.123:0/1691214802 conn(0x7f7d200fca80 msgr2=0x7f7d200feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 -- 172.21.15.123:0/1691214802 shutdown_connections 2024-03-24T00:41:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.662+0000 7f7d24a00640 1 -- 172.21.15.123:0/1691214802 wait complete. 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.663+0000 7f7d24a00640 1 Processor -- start 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.663+0000 7f7d24a00640 1 -- start start 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d20102700 0x7f7d20199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 0x7f7d201a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2010d1f0 con 0x7f7d201030c0 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2010d070 con 0x7f7d20101510 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d24a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2010d370 con 0x7f7d20102700 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37436/0 (socket says 172.21.15.123:37436) 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 -- 172.21.15.123:0/1016548840 learned_addr learned my addr 172.21.15.123:0/1016548840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d20102700 0x7f7d20199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 0x7f7d201a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 -- 172.21.15.123:0/1016548840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d20102700 msgr2=0x7f7d20199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d20102700 0x7f7d20199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 -- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 msgr2=0x7f7d201a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 0x7f7d201a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1f400640 1 -- 172.21.15.123:0/1016548840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d201a15d0 con 0x7f7d20101510 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.664+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 0x7f7d201a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d1f400640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000e9c0 tx=0x7f7d1000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d17e00640 1 -- 172.21.15.123:0/1016548840 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1000cd80 con 0x7f7d20101510 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d201a18c0 con 0x7f7d20101510 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d17e00640 1 -- 172.21.15.123:0/1016548840 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d100035f0 con 0x7f7d20101510 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d17e00640 1 -- 172.21.15.123:0/1016548840 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10010690 con 0x7f7d20101510 2024-03-24T00:41:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.665+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d201a1de0 con 0x7f7d20101510 2024-03-24T00:41:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.668+0000 7f7d17e00640 1 -- 172.21.15.123:0/1016548840 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7d100037a0 con 0x7f7d20101510 2024-03-24T00:41:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 Processor -- start 2024-03-24T00:41:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 -- start start 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6dd693b0 con 0x55cf6ddabf50 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6dd0b830 con 0x55cf6dda63f0 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6dcb45d0 con 0x55cf6dda6de0 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37440/0 (socket says 172.21.15.123:37440) 2024-03-24T00:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.669+0000 7f7d1fe00640 1 -- 172.21.15.123:0/352566537 learned_addr learned my addr 172.21.15.123:0/352566537 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 -- 172.21.15.123:0/352566537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 msgr2=0x55cf6ddab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddab8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 -- 172.21.15.123:0/352566537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 msgr2=0x55cf6ddae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddae3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 -- 172.21.15.123:0/352566537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf6dccc2c0 con 0x55cf6dda63f0 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1f400640 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009f90 tx=0x7f7d1800f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38823d537b0831e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d17e00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 0x7f7d040a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.670+0000 7f7d0be00640 1 -- 172.21.15.123:0/352566537 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18003b70 con 0x55cf6dda63f0 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d0be00640 1 -- 172.21.15.123:0/352566537 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d18003d10 con 0x55cf6dda63f0 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d0be00640 1 -- 172.21.15.123:0/352566537 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18011a30 con 0x55cf6dda63f0 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d1f400640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 0x7f7d040a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d17e00640 1 -- 172.21.15.123:0/1016548840 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7d100cef00 con 0x7f7d20101510 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d28ddc740 1 -- 172.21.15.123:0/352566537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 msgr2=0x55cf6dda6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d1f400640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 0x7f7d040a4cb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7d100ccd80 tx=0x7f7d100d29e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.671+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009f90 tx=0x7f7d1800f7f0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 -- 172.21.15.123:0/352566537 shutdown_connections 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/352566537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda63f0 0x55cf6dda6810 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 -- 172.21.15.123:0/352566537 >> 172.21.15.123:0/352566537 conn(0x55cf6dda1960 msgr2=0x55cf6dda1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 -- 172.21.15.123:0/352566537 shutdown_connections 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 -- 172.21.15.123:0/352566537 wait complete. 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 -- start start 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.672+0000 7f7d28ddc740 1 Processor -- start 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6dcc7a20 con 0x55cf6ddabf50 2024-03-24T00:41:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6dd078d0 con 0x55cf6dda6de0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6ddaef00 con 0x55cf6dda63f0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d28ddc740 10 client.0 _ll_register_callbacks cb 0x55cf6dd041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37452/0 (socket says 172.21.15.123:37452) 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1ea00640 1 -- 172.21.15.123:0/993692275 learned_addr learned my addr 172.21.15.123:0/993692275 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.673+0000 7f7d1ea00640 1 -- 172.21.15.123:0/993692275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 msgr2=0x55cf6ddb05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1ea00640 1 -- 172.21.15.123:0/993692275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 msgr2=0x55cf6ddb7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1ea00640 1 -- 172.21.15.123:0/993692275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf6da20210 con 0x55cf6dda6de0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000c00 tx=0x7f7d0c004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ef5090f569e65b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c005d50 con 0x55cf6dda6de0 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7d0c0046c0 con 0x55cf6dda6de0 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d18003d10 con 0x55cf6dda6de0 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d0c0048e0 con 0x55cf6dda6de0 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d28ddc740 10 client.25826 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.674+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cf6daec620 con 0x55cf6dda6de0 2024-03-24T00:41:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.675+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d18003d10 con 0x55cf6dda6de0 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.675+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7d0c004eb0 con 0x55cf6dda6de0 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 10 client.25826 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 10 client.25826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 20 client.25826 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6ddbc390 0x55cf6ddbe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 4 client.25826 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf6dd18d70 con 0x55cf6ddbc390 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6ddbc390 0x55cf6ddbe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d08c00640 10 client.25826 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.676+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6ddbc390 0x55cf6ddbe780 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4854631d171dce2c server_cookie=a154c6ba84359416 in_seq=0 out_seq=0 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.677+0000 7f7d08c00640 1 -- 172.21.15.123:0/993692275 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cf6dd18d70 con 0x55cf6ddbc390 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.677+0000 7f7d08c00640 10 client.25826 handle_command_reply: tid=0 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.677+0000 7f7d28ddc740 1 client.25826 shutdown 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.677+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 msgr2=0x55cf6ddb0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.677+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000c00 tx=0x7f7d0c004250 comp rx=0 tx=0).stop 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 shutdown_connections 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6ddbc390 0x55cf6ddbe780 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/993692275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 >> 172.21.15.123:0/993692275 conn(0x55cf6dda1960 msgr2=0x55cf6dda0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d08c00640 0 client.25826 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 shutdown_connections 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 1 -- 172.21.15.123:0/993692275 wait complete. 2024-03-24T00:41:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.678+0000 7f7d28ddc740 20 client.25826 trim_cache size 0 max 16384 2024-03-24T00:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 Processor -- start 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 -- start start 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201a8bd0 con 0x55cf6dda63f0 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201ab190 con 0x55cf6ddabf50 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.695+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201a91d0 con 0x55cf6dda6de0 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43586/0 (socket says 172.21.15.123:43586) 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1747284321 learned_addr learned my addr 172.21.15.123:0/1747284321 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1747284321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6ddabf50 msgr2=0x55cf6ddb7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1747284321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6dda63f0 msgr2=0x55cf6ddb05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1747284321 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf6ddb15a0 con 0x55cf6dda6de0 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.696+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c02c170 tx=0x7f7d0c02c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c500f228718296 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d0b400640 1 -- 172.21.15.123:0/1747284321 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c005d50 con 0x55cf6dda6de0 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d0b400640 1 -- 172.21.15.123:0/1747284321 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d0c004eb0 con 0x55cf6dda6de0 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d0b400640 1 -- 172.21.15.123:0/1747284321 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c003810 con 0x55cf6dda6de0 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1747284321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 msgr2=0x55cf6ddb0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c02c170 tx=0x7f7d0c02c2b0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1747284321 shutdown_connections 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb0af0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1747284321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.697+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1747284321 >> 172.21.15.123:0/1747284321 conn(0x55cf6dda1960 msgr2=0x55cf6dd9c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.698+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1747284321 shutdown_connections 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.698+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1747284321 wait complete. 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.698+0000 7f7d28ddc740 1 -- start start 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.698+0000 7f7d28ddc740 1 Processor -- start 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201a9810 con 0x55cf6ddabf50 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201aab30 con 0x55cf6dda6de0 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201a9e70 con 0x55cf6dda63f0 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d28ddc740 10 client.0 _ll_register_callbacks cb 0x7f7d0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:37482/0 (socket says 172.21.15.123:37482) 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1744461712 learned_addr learned my addr 172.21.15.123:0/1744461712 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1744461712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 msgr2=0x55cf6ddb5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1744461712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 msgr2=0x55cf6ddb6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 -- 172.21.15.123:0/1744461712 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf6ddb7020 con 0x55cf6dda6de0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.699+0000 7f7d1ea00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c004650 tx=0x7f7d0c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543b5d3d39e0c947 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c005d50 con 0x55cf6dda6de0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7d0c0037f0 con 0x55cf6dda6de0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d0c004eb0 con 0x55cf6dda6de0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d0c004250 con 0x55cf6dda6de0 2024-03-24T00:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d28ddc740 10 client.25832 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cf6de17ec0 con 0x55cf6dda6de0 2024-03-24T00:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.700+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d0c004eb0 con 0x55cf6dda6de0 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7d0c06ab30 con 0x55cf6dda6de0 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 10 client.25832 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 10 client.25832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 20 client.25832 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6de186c0 0x55cf6de1ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 4 client.25832 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cf6dd130c0 con 0x55cf6de186c0 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6de186c0 0x55cf6de1ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d1fe00640 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6de186c0 0x55cf6de1ab60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dd7bf9e876bdf94 server_cookie=c1894d8bb46223d5 in_seq=0 out_seq=0 2024-03-24T00:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:07.701+0000 7f7d08c00640 10 client.25832 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:07 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:07 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:41:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:07 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:41:08.561 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:08 smithi204 ceph-mon[36345]: pgmap v482: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 49 MiB/s wr, 5.60k op/s 2024-03-24T00:41:08.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:08 smithi123 ceph-mon[31487]: pgmap v482: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 49 MiB/s wr, 5.60k op/s 2024-03-24T00:41:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:08 smithi177 ceph-mon[36034]: pgmap v482: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 49 MiB/s wr, 5.60k op/s 2024-03-24T00:41:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:10 smithi204 ceph-mon[36345]: pgmap v483: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 59 MiB/s wr, 6.82k op/s 2024-03-24T00:41:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:10 smithi123 ceph-mon[31487]: pgmap v483: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 59 MiB/s wr, 6.82k op/s 2024-03-24T00:41:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:10 smithi177 ceph-mon[36034]: pgmap v483: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 59 MiB/s wr, 6.82k op/s 2024-03-24T00:41:12.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:12 smithi204 ceph-mon[36345]: pgmap v484: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 63 MiB/s wr, 7.37k op/s 2024-03-24T00:41:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:12 smithi123 ceph-mon[31487]: pgmap v484: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 63 MiB/s wr, 7.37k op/s 2024-03-24T00:41:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:12 smithi177 ceph-mon[36034]: pgmap v484: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 63 MiB/s wr, 7.37k op/s 2024-03-24T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:14 smithi204 ceph-mon[36345]: pgmap v485: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 58 MiB/s wr, 6.82k op/s 2024-03-24T00:41:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:14 smithi123 ceph-mon[31487]: pgmap v485: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 58 MiB/s wr, 6.82k op/s 2024-03-24T00:41:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:14 smithi177 ceph-mon[36034]: pgmap v485: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 58 MiB/s wr, 6.82k op/s 2024-03-24T00:41:16.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:16 smithi204 ceph-mon[36345]: pgmap v486: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 59 MiB/s wr, 7.10k op/s 2024-03-24T00:41:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:16 smithi123 ceph-mon[31487]: pgmap v486: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 59 MiB/s wr, 7.10k op/s 2024-03-24T00:41:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:16 smithi177 ceph-mon[36034]: pgmap v486: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 59 MiB/s wr, 7.10k op/s 2024-03-24T00:41:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:18 smithi123 ceph-mon[31487]: pgmap v487: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 52 MiB/s wr, 6.30k op/s 2024-03-24T00:41:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:18 smithi177 ceph-mon[36034]: pgmap v487: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 52 MiB/s wr, 6.30k op/s 2024-03-24T00:41:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.782+0000 7f7d08c00640 1 -- 172.21.15.123:0/1744461712 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cf6dd130c0 con 0x55cf6de186c0 2024-03-24T00:41:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.782+0000 7f7d08c00640 10 client.25832 handle_command_reply: tid=0 2024-03-24T00:41:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.782+0000 7f7d28ddc740 1 client.25832 shutdown 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 msgr2=0x55cf6ddb5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c004650 tx=0x7f7d0c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 shutdown_connections 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cf6ddabf50 0x55cf6ddb6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf6dda6de0 0x55cf6ddb5ac0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cf6de186c0 0x55cf6de1ab60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 --2- 172.21.15.123:0/1744461712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf6dda63f0 0x55cf6ddb5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 >> 172.21.15.123:0/1744461712 conn(0x55cf6dda1960 msgr2=0x55cf6dda4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.783+0000 7f7d08c00640 0 client.25832 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:18.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.784+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 shutdown_connections 2024-03-24T00:41:18.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.784+0000 7f7d28ddc740 1 -- 172.21.15.123:0/1744461712 wait complete. 2024-03-24T00:41:18.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.784+0000 7f7d28ddc740 20 client.25832 trim_cache size 0 max 16384 2024-03-24T00:41:18.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:41:18.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:41:18.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:41:18.786 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 msgr2=0x7f7d040a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 0x7f7d040a4cb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7d100ccd80 tx=0x7f7d100d29e0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 msgr2=0x7f7d201999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000e9c0 tx=0x7f7d1000ee80 comp rx=0 tx=0).stop 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 shutdown_connections 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7d040a27f0 0x7f7d040a4cb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d201030c0 0x7f7d201a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d20102700 0x7f7d20199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 --2- 172.21.15.123:0/1016548840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d20101510 0x7f7d201999b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.786+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 >> 172.21.15.123:0/1016548840 conn(0x7f7d200fca80 msgr2=0x7f7d200738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.787+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 shutdown_connections 2024-03-24T00:41:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:18.787+0000 7f7d24a00640 1 -- 172.21.15.123:0/1016548840 wait complete. 2024-03-24T00:41:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:18 smithi204 ceph-mon[36345]: pgmap v487: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 52 MiB/s wr, 6.30k op/s 2024-03-24T00:41:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:20 smithi123 ceph-mon[31487]: pgmap v488: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 58 MiB/s wr, 7.11k op/s 2024-03-24T00:41:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:20 smithi177 ceph-mon[36034]: pgmap v488: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 58 MiB/s wr, 7.11k op/s 2024-03-24T00:41:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:20 smithi204 ceph-mon[36345]: pgmap v488: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 58 MiB/s wr, 7.11k op/s 2024-03-24T00:41:22.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:22 smithi123 ceph-mon[31487]: pgmap v489: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 61 MiB/s wr, 7.30k op/s 2024-03-24T00:41:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:22 smithi177 ceph-mon[36034]: pgmap v489: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 61 MiB/s wr, 7.30k op/s 2024-03-24T00:41:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:22 smithi204 ceph-mon[36345]: pgmap v489: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 61 MiB/s wr, 7.30k op/s 2024-03-24T00:41:23.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:23 smithi177 ceph-mon[36034]: pgmap v490: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 54 MiB/s wr, 6.32k op/s 2024-03-24T00:41:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:23 smithi204 ceph-mon[36345]: pgmap v490: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 54 MiB/s wr, 6.32k op/s 2024-03-24T00:41:23.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:23 smithi123 ceph-mon[31487]: pgmap v490: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 54 MiB/s wr, 6.32k op/s 2024-03-24T00:41:26.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:26 smithi204 ceph-mon[36345]: pgmap v491: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 53 MiB/s wr, 6.04k op/s 2024-03-24T00:41:26.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:26 smithi123 ceph-mon[31487]: pgmap v491: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 53 MiB/s wr, 6.04k op/s 2024-03-24T00:41:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:26 smithi177 ceph-mon[36034]: pgmap v491: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 53 MiB/s wr, 6.04k op/s 2024-03-24T00:41:27.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:41:27.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:41:27.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 -- 172.21.15.123:0/1408324812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510108db0 msgr2=0x7f2510109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 --2- 172.21.15.123:0/1408324812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510108db0 0x7f2510109230 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009900 tx=0x7f24fc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 -- 172.21.15.123:0/1408324812 shutdown_connections 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 --2- 172.21.15.123:0/1408324812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510109770 0x7f2510110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 --2- 172.21.15.123:0/1408324812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510108db0 0x7f2510109230 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 --2- 172.21.15.123:0/1408324812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510107bc0 0x7f2510107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 -- 172.21.15.123:0/1408324812 >> 172.21.15.123:0/1408324812 conn(0x7f2510076b70 msgr2=0x7f2510078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 -- 172.21.15.123:0/1408324812 shutdown_connections 2024-03-24T00:41:27.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.416+0000 7f2516400640 1 -- 172.21.15.123:0/1408324812 wait complete. 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.417+0000 7f2516400640 1 Processor -- start 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 -- start start 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 0x7f251019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 0x7f25101a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2510113820 con 0x7f2510108db0 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101136a0 con 0x7f2510109770 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2516400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101139a0 con 0x7f2510107bc0 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2515000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2515000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56718/0 (socket says 172.21.15.123:56718) 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2515000640 1 -- 172.21.15.123:0/3593486283 learned_addr learned my addr 172.21.15.123:0/3593486283 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.418+0000 7f2515a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 0x7f25101a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f250fe00640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 0x7f251019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 -- 172.21.15.123:0/3593486283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 msgr2=0x7f25101a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 0x7f25101a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 -- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 msgr2=0x7f251019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 0x7f251019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 -- 172.21.15.123:0/3593486283 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25101a5ac0 con 0x7f2510107bc0 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515a00640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 0x7f25101a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:27.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f250fe00640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 0x7f251019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.419+0000 7f2515000640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f250400e9c0 tx=0x7f250400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.420+0000 7f250d600640 1 -- 172.21.15.123:0/3593486283 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250400cd80 con 0x7f2510107bc0 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.420+0000 7f250d600640 1 -- 172.21.15.123:0/3593486283 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25040035f0 con 0x7f2510107bc0 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.420+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101a5db0 con 0x7f2510107bc0 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.420+0000 7f250d600640 1 -- 172.21.15.123:0/3593486283 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504010690 con 0x7f2510107bc0 2024-03-24T00:41:27.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.420+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101a6270 con 0x7f2510107bc0 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.424+0000 7f250d600640 1 -- 172.21.15.123:0/3593486283 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2504004020 con 0x7f2510107bc0 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 Processor -- start 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 -- start start 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef393540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 0x560bef396040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef2f3250 con 0x560bef38ea40 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef29ba30 con 0x560bef393bb0 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f251a7c1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef2ef2f0 con 0x560bef38e050 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f2515a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f2515a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56734/0 (socket says 172.21.15.123:56734) 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f2515a00640 1 -- 172.21.15.123:0/1104193355 learned_addr learned my addr 172.21.15.123:0/1104193355 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f250fe00640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 0x560bef396040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.425+0000 7f2515000640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef393540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 -- 172.21.15.123:0/1104193355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 msgr2=0x560bef396040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 0x560bef396040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 -- 172.21.15.123:0/1104193355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 msgr2=0x560bef393540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef393540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 -- 172.21.15.123:0/1104193355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bef286690 con 0x560bef38e050 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f250fe00640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 0x560bef396040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2515a00640 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25080107e0 tx=0x7f2508010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cee1cf6e5a6ee624 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f250d600640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 0x7f24f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2502000640 1 -- 172.21.15.123:0/1104193355 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508003b70 con 0x560bef38e050 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2502000640 1 -- 172.21.15.123:0/1104193355 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f250800ee10 con 0x560bef38e050 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.426+0000 7f2502000640 1 -- 172.21.15.123:0/1104193355 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080047a0 con 0x560bef38e050 2024-03-24T00:41:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.427+0000 7f250d600640 1 -- 172.21.15.123:0/3593486283 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f25040cee60 con 0x7f2510107bc0 2024-03-24T00:41:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.427+0000 7f2515000640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 0x7f24f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.427+0000 7f251a7c1740 1 -- 172.21.15.123:0/1104193355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 msgr2=0x560bef38e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.427+0000 7f251a7c1740 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25080107e0 tx=0x7f2508010ca0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f2515000640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 0x7f24f80a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f25040ccce0 tx=0x7f2504000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 -- 172.21.15.123:0/1104193355 shutdown_connections 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef393bb0 0x560bef396040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef393540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 --2- 172.21.15.123:0/1104193355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef38e470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 -- 172.21.15.123:0/1104193355 >> 172.21.15.123:0/1104193355 conn(0x560bef356200 msgr2=0x560bef3899c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 -- 172.21.15.123:0/1104193355 shutdown_connections 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 -- 172.21.15.123:0/1104193355 wait complete. 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.428+0000 7f251a7c1740 1 -- start start 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 Processor -- start 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef398080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 0x560bef3985c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef396b60 con 0x560bef393bb0 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef2aee80 con 0x560bef38ea40 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bef396ce0 con 0x560bef38e050 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.429+0000 7f251a7c1740 10 client.0 _ll_register_callbacks cb 0x560bef2c1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f2515a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef398080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f2515a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 0x560bef3985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f2515a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef398080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56740/0 (socket says 172.21.15.123:56740) 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f2515a00640 1 -- 172.21.15.123:0/2395778347 learned_addr learned my addr 172.21.15.123:0/2395778347 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 -- 172.21.15.123:0/2395778347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 msgr2=0x560bef398080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef398080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 -- 172.21.15.123:0/2395778347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 msgr2=0x560bef39f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.430+0000 7f250fe00640 1 -- 172.21.15.123:0/2395778347 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bef0d17d0 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f250fe00640 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 0x560bef3985c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1adfe0745492dc65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f24f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc005d50 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f24fc005220 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f250800ee10 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24fc04c050 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f251a7c1740 10 client.25844 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.431+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560bef0073f0 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f250800ee10 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f24fc003940 con 0x560bef38ea40 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 10 client.25844 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 10 client.25844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 20 client.25844 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef3a3e70 0x560bef3a6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 4 client.25844 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560bef300790 con 0x560bef3a3e70 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.432+0000 7f2515a00640 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef3a3e70 0x560bef3a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.433+0000 7f2515a00640 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef3a3e70 0x560bef3a6260 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5ac42b38a1650d server_cookie=208074f320fcd92 in_seq=0 out_seq=0 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.433+0000 7f24f6a00640 10 client.25844 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.433+0000 7f24f6a00640 1 -- 172.21.15.123:0/2395778347 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560bef300790 con 0x560bef3a3e70 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f24f6a00640 10 client.25844 handle_command_reply: tid=0 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 client.25844 shutdown 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 msgr2=0x560bef3985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 0x560bef3985c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc004270 comp rx=0 tx=0).stop 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 shutdown_connections 2024-03-24T00:41:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38ea40 0x560bef3985c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef3a3e70 0x560bef3a6260 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2395778347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38e050 0x560bef398080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f24f6a00640 0 client.25844 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 >> 172.21.15.123:0/2395778347 conn(0x560bef356200 msgr2=0x560bef388bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 shutdown_connections 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 1 -- 172.21.15.123:0/2395778347 wait complete. 2024-03-24T00:41:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.434+0000 7f251a7c1740 20 client.25844 trim_cache size 0 max 16384 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.454+0000 7f251a7c1740 1 Processor -- start 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.454+0000 7f251a7c1740 1 -- start start 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef39f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef393bb0 0x560bef38d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101acec0 con 0x560bef38ea40 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101af480 con 0x560bef38e050 2024-03-24T00:41:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f251a7c1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101ad4c0 con 0x560bef393bb0 2024-03-24T00:41:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f2515a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f250fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef39f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f2515a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56418/0 (socket says 172.21.15.123:56418) 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f2515a00640 1 -- 172.21.15.123:0/508450893 learned_addr learned my addr 172.21.15.123:0/508450893 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.455+0000 7f2515a00640 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef393bb0 0x560bef38d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 -- 172.21.15.123:0/508450893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef393bb0 msgr2=0x560bef38d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef393bb0 0x560bef38d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 -- 172.21.15.123:0/508450893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 msgr2=0x560bef39f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef39f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 -- 172.21.15.123:0/508450893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560beefc70f0 con 0x560bef38e050 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2515a00640 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2508010700 tx=0x7f25080096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f4cfe5a726d4b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2501600640 1 -- 172.21.15.123:0/508450893 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508025040 con 0x560bef38e050 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2501600640 1 -- 172.21.15.123:0/508450893 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f250801e020 con 0x560bef38e050 2024-03-24T00:41:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.456+0000 7f2501600640 1 -- 172.21.15.123:0/508450893 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080052b0 con 0x560bef38e050 2024-03-24T00:41:27.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 -- 172.21.15.123:0/508450893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 msgr2=0x560bef38cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2508010700 tx=0x7f25080096a0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 -- 172.21.15.123:0/508450893 shutdown_connections 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef393bb0 0x560bef38d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef38ea40 0x560bef39f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 --2- 172.21.15.123:0/508450893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef38cc00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 -- 172.21.15.123:0/508450893 >> 172.21.15.123:0/508450893 conn(0x560bef356200 msgr2=0x560bef38fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 -- 172.21.15.123:0/508450893 shutdown_connections 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.457+0000 7f251a7c1740 1 -- 172.21.15.123:0/508450893 wait complete. 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.458+0000 7f251a7c1740 1 -- start start 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.458+0000 7f251a7c1740 1 Processor -- start 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.458+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef39d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.458+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.458+0000 7f251a7c1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f251a7c1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101adb00 con 0x560bef393bb0 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f251a7c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101aee20 con 0x560bef38e050 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f251a7c1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101ae160 con 0x560bef38ea40 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f251a7c1740 10 client.0 _ll_register_callbacks cb 0x7f24fc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56766/0 (socket says 172.21.15.123:56766) 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 -- 172.21.15.123:0/2326878934 learned_addr learned my addr 172.21.15.123:0/2326878934 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f2515a00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef39d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f2515a00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 -- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 msgr2=0x560bef39d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef39d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 -- 172.21.15.123:0/2326878934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 msgr2=0x560bef39e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 -- 172.21.15.123:0/2326878934 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bef39edd0 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f2515a00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef39d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f250fe00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24fc02c7b0 tx=0x7f24fc003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=430cea687db08715 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.459+0000 7f24f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc000e60 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f250801e020 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f24fc034830 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24fc005d50 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 10 client.25846 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560bef3ffd40 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f250801e020 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f24fc058030 con 0x560bef38ea40 2024-03-24T00:41:27.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 10 client.25846 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 10 client.25846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.460+0000 7f251a7c1740 20 client.25846 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef4006b0 0x560bef402ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f251a7c1740 4 client.25846 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560bef2faae0 con 0x560bef4006b0 2024-03-24T00:41:27.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f2515a00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef4006b0 0x560bef402ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f2515a00640 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef4006b0 0x560bef402ad0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1122a7552fa530cc server_cookie=69150c14a545af75 in_seq=0 out_seq=0 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.461+0000 7f24f6a00640 10 client.25846 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f24f6a00640 1 -- 172.21.15.123:0/2326878934 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560bef2faae0 con 0x560bef4006b0 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f24f6a00640 10 client.25846 handle_command_reply: tid=0 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 client.25846 shutdown 2024-03-24T00:41:27.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 msgr2=0x560bef39d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24fc02c7b0 tx=0x7f24fc003820 comp rx=0 tx=0).stop 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 shutdown_connections 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560bef393bb0 0x560bef39e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560bef4006b0 0x560bef402ad0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bef38ea40 0x560bef39d820 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 --2- 172.21.15.123:0/2326878934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560bef38e050 0x560bef39d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 >> 172.21.15.123:0/2326878934 conn(0x560bef356200 msgr2=0x560bef38f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f24f6a00640 0 client.25846 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 shutdown_connections 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 1 -- 172.21.15.123:0/2326878934 wait complete. 2024-03-24T00:41:27.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.462+0000 7f251a7c1740 20 client.25846 trim_cache size 0 max 16384 2024-03-24T00:41:27.468 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:41:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:41:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:41:27.468 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 msgr2=0x7f24f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 0x7f24f80a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f25040ccce0 tx=0x7f2504000bc0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 msgr2=0x7f251019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f250400e9c0 tx=0x7f250400ee80 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 shutdown_connections 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2510109770 0x7f25101a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f24f80a2880 0x7f24f80a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2510108db0 0x7f251019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 --2- 172.21.15.123:0/3593486283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2510107bc0 0x7f251019dea0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 >> 172.21.15.123:0/3593486283 conn(0x7f2510076b70 msgr2=0x7f251010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 shutdown_connections 2024-03-24T00:41:27.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.463+0000 7f2516400640 1 -- 172.21.15.123:0/3593486283 wait complete. 2024-03-24T00:41:27.483 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:41:27.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 -- 172.21.15.123:0/2892405511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c102010 msgr2=0x7feb7c104400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 --2- 172.21.15.123:0/2892405511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c102010 0x7feb7c104400 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009900 tx=0x7feb6c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 -- 172.21.15.123:0/2892405511 shutdown_connections 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 --2- 172.21.15.123:0/2892405511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c107270 0x7feb7c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 --2- 172.21.15.123:0/2892405511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 0x7feb7c106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 --2- 172.21.15.123:0/2892405511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c102010 0x7feb7c104400 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 -- 172.21.15.123:0/2892405511 >> 172.21.15.123:0/2892405511 conn(0x7feb7c0fb980 msgr2=0x7feb7c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 -- 172.21.15.123:0/2892405511 shutdown_connections 2024-03-24T00:41:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.687+0000 7feb7b200640 1 -- 172.21.15.123:0/2892405511 wait complete. 2024-03-24T00:41:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.688+0000 7feb7b200640 1 Processor -- start 2024-03-24T00:41:27.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 -- start start 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 0x7feb7c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 0x7feb7c1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c10d1f0 con 0x7feb7c104940 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c10d070 con 0x7feb7c107270 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c10d370 con 0x7feb7c102010 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb79e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb79e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56784/0 (socket says 172.21.15.123:56784) 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb79e00640 1 -- 172.21.15.123:0/1602422626 learned_addr learned my addr 172.21.15.123:0/1602422626 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb7a800640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 0x7feb7c1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb79400640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 0x7feb7c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.689+0000 7feb79e00640 1 -- 172.21.15.123:0/1602422626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 msgr2=0x7feb7c1a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb79e00640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 0x7feb7c1a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb79e00640 1 -- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 msgr2=0x7feb7c199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb79e00640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 0x7feb7c199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb79e00640 1 -- 172.21.15.123:0/1602422626 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c1a1620 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb7a800640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 0x7feb7c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb79e00640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c043020 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb6c0054e0 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c003c40 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.690+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c1a18b0 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.691+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c1a1db0 con 0x7feb7c102010 2024-03-24T00:41:27.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.692+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c108d60 con 0x7feb7c102010 2024-03-24T00:41:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.694+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feb6c046020 con 0x7feb7c102010 2024-03-24T00:41:27.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.696+0000 7feb72a00640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 0x7feb600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.696+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feb6c0f0aa0 con 0x7feb7c102010 2024-03-24T00:41:27.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.696+0000 7feb79400640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 0x7feb600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.697+0000 7feb79400640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 0x7feb600a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feb7c19af10 tx=0x7feb68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:27.700+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb6c04d020 con 0x7feb7c102010 2024-03-24T00:41:28.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.044+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb7c0622a0 con 0x7feb7c102010 2024-03-24T00:41:28.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.045+0000 7feb72a00640 1 -- 172.21.15.123:0/1602422626 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7feb6c0bdaf0 con 0x7feb7c102010 2024-03-24T00:41:28.046 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:41:28.047 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:41:28.047 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:41:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.050+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 msgr2=0x7feb600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.050+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 0x7feb600a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feb7c19af10 tx=0x7feb68005f90 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.050+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 msgr2=0x7feb7c199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.050+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c02c650 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 shutdown_connections 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7c107270 0x7feb7c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feb600a2880 0x7feb600a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7c104940 0x7feb7c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 --2- 172.21.15.123:0/1602422626 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb7c102010 0x7feb7c199a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 >> 172.21.15.123:0/1602422626 conn(0x7feb7c0fb980 msgr2=0x7feb7c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 shutdown_connections 2024-03-24T00:41:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.051+0000 7feb7b200640 1 -- 172.21.15.123:0/1602422626 wait complete. 2024-03-24T00:41:28.074 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:41:28.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:41:28.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:28 smithi123 ceph-mon[31487]: pgmap v492: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 49 MiB/s wr, 5.49k op/s 2024-03-24T00:41:28.356 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:28 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1602422626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:41:28.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 -- 172.21.15.123:0/225309083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40081073e0 msgr2=0x7f40081097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 --2- 172.21.15.123:0/225309083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40081073e0 0x7f40081097d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4000009ca0 tx=0x7f400002c100 comp rx=0 tx=0).stop 2024-03-24T00:41:28.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 -- 172.21.15.123:0/225309083 shutdown_connections 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 --2- 172.21.15.123:0/225309083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40081073e0 0x7f40081097d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 --2- 172.21.15.123:0/225309083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008104ab0 0x7f4008106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 --2- 172.21.15.123:0/225309083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4008102180 0x7f4008104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.382+0000 7f4010200640 1 -- 172.21.15.123:0/225309083 >> 172.21.15.123:0/225309083 conn(0x7f40080fbb30 msgr2=0x7f40080fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.383+0000 7f4010200640 1 -- 172.21.15.123:0/225309083 shutdown_connections 2024-03-24T00:41:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.383+0000 7f4010200640 1 -- 172.21.15.123:0/225309083 wait complete. 2024-03-24T00:41:28.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.385+0000 7f4010200640 1 Processor -- start 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.385+0000 7f4010200640 1 -- start start 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4008104ab0 0x7f4008110fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 0x7f4008118010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4008101520 con 0x7f4008104ab0 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081013a0 con 0x7f40081073e0 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f4010200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081016a0 con 0x7f4008102180 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56806/0 (socket says 172.21.15.123:56806) 2024-03-24T00:41:28.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.386+0000 7f400d800640 1 -- 172.21.15.123:0/1546296796 learned_addr learned my addr 172.21.15.123:0/1546296796 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 -- 172.21.15.123:0/1546296796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 msgr2=0x7f4008118010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400e200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 0x7f4008118010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 0x7f4008118010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 -- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4008104ab0 msgr2=0x7f4008110fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4008104ab0 0x7f4008110fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 -- 172.21.15.123:0/1546296796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008118680 con 0x7f4008102180 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400e200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 0x7f4008118010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f400d800640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00e9f0 tx=0x7f3ffc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.387+0000 7f4006000640 1 -- 172.21.15.123:0/1546296796 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00cd80 con 0x7f4008102180 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.388+0000 7f4006000640 1 -- 172.21.15.123:0/1546296796 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc010910 con 0x7f4008102180 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.388+0000 7f4006000640 1 -- 172.21.15.123:0/1546296796 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00cd80 con 0x7f4008102180 2024-03-24T00:41:28.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.388+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40081ae6b0 con 0x7f4008102180 2024-03-24T00:41:28.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.389+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40081aeaf0 con 0x7f4008102180 2024-03-24T00:41:28.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.394+0000 7f4013017740 1 Processor -- start 2024-03-24T00:41:28.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.394+0000 7f4006000640 1 -- 172.21.15.123:0/1546296796 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3ffc010ac0 con 0x7f4008102180 2024-03-24T00:41:28.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 -- start start 2024-03-24T00:41:28.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4006000640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 0x7f3ff00a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bb90e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb97c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976baf42c0 con 0x55976bb913a0 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976ba9cc90 con 0x55976bb909e0 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.395+0000 7f4013017740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976baf0360 con 0x55976bb8f810 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.396+0000 7f4006000640 1 -- 172.21.15.123:0/1546296796 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ffc0cebb0 con 0x7f4008102180 2024-03-24T00:41:28.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.396+0000 7f400ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bb90e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.396+0000 7f400e200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 0x7f3ff00a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.397+0000 7f400ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bb90e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56464/0 (socket says 172.21.15.123:56464) 2024-03-24T00:41:28.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.397+0000 7f400ce00640 1 -- 172.21.15.123:0/3204230998 learned_addr learned my addr 172.21.15.123:0/3204230998 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:28.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.397+0000 7f400d800640 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400e200640 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb97c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400d800640 1 -- 172.21.15.123:0/3204230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 msgr2=0x55976bb90e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400d800640 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bb90e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400d800640 1 -- 172.21.15.123:0/3204230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 msgr2=0x55976bb97c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400d800640 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb97c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.398+0000 7f400d800640 1 -- 172.21.15.123:0/3204230998 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55976bac1080 con 0x55976bb8f810 2024-03-24T00:41:28.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.399+0000 7f400d800640 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8fc30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc09b200 tx=0x7f3ffc09b400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37fa78855a13e3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.400+0000 7f3ffaa00640 1 -- 172.21.15.123:0/3204230998 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0a3ca0 con 0x55976bb8f810 2024-03-24T00:41:28.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.400+0000 7f3ffaa00640 1 -- 172.21.15.123:0/3204230998 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc099800 con 0x55976bb8f810 2024-03-24T00:41:28.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.400+0000 7f3ffaa00640 1 -- 172.21.15.123:0/3204230998 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc09a950 con 0x55976bb8f810 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 -- 172.21.15.123:0/3204230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 msgr2=0x55976bb8fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8fc30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc09b200 tx=0x7f3ffc09b400 comp rx=0 tx=0).stop 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f400e200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 0x7f3ff00a4fc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4000005e30 tx=0x7f4000009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 -- 172.21.15.123:0/3204230998 shutdown_connections 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb97c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bb90e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 --2- 172.21.15.123:0/3204230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8fc30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 -- 172.21.15.123:0/3204230998 >> 172.21.15.123:0/3204230998 conn(0x55976bb8ac00 msgr2=0x55976bb8b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.401+0000 7f4013017740 1 -- 172.21.15.123:0/3204230998 shutdown_connections 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.402+0000 7f4013017740 1 -- 172.21.15.123:0/3204230998 wait complete. 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.402+0000 7f4013017740 1 -- start start 2024-03-24T00:41:28.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.403+0000 7f4013017740 1 Processor -- start 2024-03-24T00:41:28.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f4013017740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f4013017740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb909e0 0x55976bb9a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56816/0 (socket says 172.21.15.123:56816) 2024-03-24T00:41:28.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f400d800640 1 -- 172.21.15.123:0/686001475 learned_addr learned my addr 172.21.15.123:0/686001475 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.404+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.405+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976bb983b0 con 0x55976bb909e0 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.405+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976bab00d0 con 0x55976bb913a0 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.405+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976bb98530 con 0x55976bb8f810 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.405+0000 7f400e200640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.405+0000 7f4013017740 10 client.0 _ll_register_callbacks cb 0x55976bab4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400ce00640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb909e0 0x55976bb9a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400d800640 1 -- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 msgr2=0x55976bba1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400d800640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400d800640 1 -- 172.21.15.123:0/686001475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb909e0 msgr2=0x55976bb9a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400d800640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb909e0 0x55976bb9a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.406+0000 7f400d800640 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55976b8083f0 con 0x55976bb8f810 2024-03-24T00:41:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.408+0000 7f400d800640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc01e000 tx=0x7f3ffc0cc670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb97bd86e00015c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.408+0000 7f3fef400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:41:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.408+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b3030 con 0x55976bb8f810 2024-03-24T00:41:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.409+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ffc0995a0 con 0x55976bb8f810 2024-03-24T00:41:28.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.409+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ffc099800 con 0x55976bb8f810 2024-03-24T00:41:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.411+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ffc09ab60 con 0x55976bb8f810 2024-03-24T00:41:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.411+0000 7f4013017740 10 client.25864 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.411+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55976b8d27d0 con 0x55976bb8f810 2024-03-24T00:41:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.412+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc099800 con 0x55976bb8f810 2024-03-24T00:41:28.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3ffc097250 con 0x55976bb8f810 2024-03-24T00:41:28.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 10 client.25864 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 10 client.25864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 20 client.25864 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bba51c0 0x55976bba75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 4 client.25864 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.413+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55976bb01800 con 0x55976bba51c0 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.414+0000 7f400ce00640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bba51c0 0x55976bba75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.415+0000 7f400ce00640 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bba51c0 0x55976bba75b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0135b18ed393417 server_cookie=ace468e2dbb5bc8 in_seq=0 out_seq=0 2024-03-24T00:41:28.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.415+0000 7f3fef400640 10 client.25864 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:28.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.416+0000 7f3fef400640 1 -- 172.21.15.123:0/686001475 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55976bb01800 con 0x55976bba51c0 2024-03-24T00:41:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.416+0000 7f3fef400640 10 client.25864 handle_command_reply: tid=0 2024-03-24T00:41:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.416+0000 7f4013017740 1 client.25864 shutdown 2024-03-24T00:41:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 msgr2=0x55976bb99ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc01e000 tx=0x7f3ffc0cc670 comp rx=0 tx=0).stop 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 shutdown_connections 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb909e0 0x55976bb9a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bba51c0 0x55976bba75b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 --2- 172.21.15.123:0/686001475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb99ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 >> 172.21.15.123:0/686001475 conn(0x55976bb8ac00 msgr2=0x55976bb89eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.417+0000 7f3fef400640 0 client.25864 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.418+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 shutdown_connections 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.418+0000 7f4013017740 1 -- 172.21.15.123:0/686001475 wait complete. 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.418+0000 7f4013017740 20 client.25864 trim_cache size 0 max 16384 2024-03-24T00:41:28.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.446+0000 7f4013017740 1 Processor -- start 2024-03-24T00:41:28.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.446+0000 7f4013017740 1 -- start start 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bba1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb93b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b59c0 con 0x55976bb913a0 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b7f80 con 0x55976bb909e0 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f4013017740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b5fc0 con 0x55976bb8f810 2024-03-24T00:41:28.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f400e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb93b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f400d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56828/0 (socket says 172.21.15.123:56828) 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f400d800640 1 -- 172.21.15.123:0/1547850055 learned_addr learned my addr 172.21.15.123:0/1547850055 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.447+0000 7f400ce00640 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bba1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 -- 172.21.15.123:0/1547850055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 msgr2=0x55976bba1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bba1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 -- 172.21.15.123:0/1547850055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 msgr2=0x55976bb93b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb93b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 -- 172.21.15.123:0/1547850055 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55976bb975e0 con 0x55976bb8f810 2024-03-24T00:41:28.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f400d800640 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc097650 tx=0x7f3ffc09b400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4e6e57b522efce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.448+0000 7f3ffa000640 1 -- 172.21.15.123:0/1547850055 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0ab070 con 0x55976bb8f810 2024-03-24T00:41:28.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.449+0000 7f3ffa000640 1 -- 172.21.15.123:0/1547850055 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc099800 con 0x55976bb8f810 2024-03-24T00:41:28.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.449+0000 7f3ffa000640 1 -- 172.21.15.123:0/1547850055 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b3070 con 0x55976bb8f810 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- 172.21.15.123:0/1547850055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 msgr2=0x55976bb8afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc097650 tx=0x7f3ffc09b400 comp rx=0 tx=0).stop 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- 172.21.15.123:0/1547850055 shutdown_connections 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb913a0 0x55976bb93b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb909e0 0x55976bba1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 --2- 172.21.15.123:0/1547850055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb8f810 0x55976bb8afe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- 172.21.15.123:0/1547850055 >> 172.21.15.123:0/1547850055 conn(0x55976bb8ac00 msgr2=0x55976bb8e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- 172.21.15.123:0/1547850055 shutdown_connections 2024-03-24T00:41:28.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- 172.21.15.123:0/1547850055 wait complete. 2024-03-24T00:41:28.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.450+0000 7f4013017740 1 -- start start 2024-03-24T00:41:28.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.451+0000 7f4013017740 1 Processor -- start 2024-03-24T00:41:28.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.451+0000 7f4013017740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb8f810 0x55976bb9d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.451+0000 7f4013017740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.451+0000 7f400ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f4013017740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f4013017740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b6600 con 0x55976bb8f810 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f4013017740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b7920 con 0x55976bb913a0 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f4013017740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081b6c60 con 0x55976bb909e0 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f4013017740 10 client.0 _ll_register_callbacks cb 0x7f3ff4009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f400ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56830/0 (socket says 172.21.15.123:56830) 2024-03-24T00:41:28.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f400ce00640 1 -- 172.21.15.123:0/4164830639 learned_addr learned my addr 172.21.15.123:0/4164830639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f400d800640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb8f810 0x55976bb9d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.452+0000 7f400e200640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 -- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 msgr2=0x55976bba0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 -- 172.21.15.123:0/4164830639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb8f810 msgr2=0x55976bb9d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb8f810 0x55976bb9d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 -- 172.21.15.123:0/4164830639 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55976bba0af0 con 0x55976bb909e0 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f400ce00640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4005fd0 tx=0x7f3ff401b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5175d32d5720adcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f3fef400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:41:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4012650 con 0x55976bb909e0 2024-03-24T00:41:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ff40127f0 con 0x55976bb909e0 2024-03-24T00:41:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.453+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ffc099800 con 0x55976bb909e0 2024-03-24T00:41:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.454+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ff40129f0 con 0x55976bb909e0 2024-03-24T00:41:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.454+0000 7f4013017740 10 client.25870 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.454+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55976bc011a0 con 0x55976bb909e0 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.454+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc099800 con 0x55976bb909e0 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3ff4011440 con 0x55976bb909e0 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 10 client.25870 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 10 client.25870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 20 client.25870 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bc01bd0 0x55976bc03ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 4 client.25870 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:41:28.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55976bafbb50 con 0x55976bc01bd0 2024-03-24T00:41:28.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.455+0000 7f400d800640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bc01bd0 0x55976bc03ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.456+0000 7f400d800640 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bc01bd0 0x55976bc03ff0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d44ca98a98ee4bca server_cookie=28494334a291f74c in_seq=0 out_seq=0 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.456+0000 7f3fef400640 10 client.25870 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.456+0000 7f3fef400640 1 -- 172.21.15.123:0/4164830639 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55976bafbb50 con 0x55976bc01bd0 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.456+0000 7f3fef400640 10 client.25870 handle_command_reply: tid=0 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.456+0000 7f4013017740 1 client.25870 shutdown 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 msgr2=0x55976bb9d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4005fd0 tx=0x7f3ff401b960 comp rx=0 tx=0).stop 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 shutdown_connections 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976bb913a0 0x55976bba0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55976bc01bd0 0x55976bc03ff0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55976bb909e0 0x55976bb9d550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 --2- 172.21.15.123:0/4164830639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55976bb8f810 0x55976bb9d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 >> 172.21.15.123:0/4164830639 conn(0x55976bb8ac00 msgr2=0x55976bb8e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f3fef400640 0 client.25870 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 shutdown_connections 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 1 -- 172.21.15.123:0/4164830639 wait complete. 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.457+0000 7f4013017740 20 client.25870 trim_cache size 0 max 16384 2024-03-24T00:41:28.460 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:41:28.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.458+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 msgr2=0x7f3ff00a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.458+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 0x7f3ff00a4fc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4000005e30 tx=0x7f4000009360 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.458+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 msgr2=0x7f4008110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.458+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00e9f0 tx=0x7f3ffc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 shutdown_connections 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40081073e0 0x7f4008118010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ff00a2b00 0x7f3ff00a4fc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4008104ab0 0x7f4008110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 --2- 172.21.15.123:0/1546296796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4008102180 0x7f4008110a60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.459+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 >> 172.21.15.123:0/1546296796 conn(0x7f40080fbb30 msgr2=0x7f4008102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.460+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 shutdown_connections 2024-03-24T00:41:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:28.460+0000 7f4010200640 1 -- 172.21.15.123:0/1546296796 wait complete. 2024-03-24T00:41:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:28 smithi204 ceph-mon[36345]: pgmap v492: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 49 MiB/s wr, 5.49k op/s 2024-03-24T00:41:28.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:28 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1602422626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:41:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:28 smithi177 ceph-mon[36034]: pgmap v492: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 49 MiB/s wr, 5.49k op/s 2024-03-24T00:41:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:28 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1602422626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:41:29.484 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:41:29.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:29.484 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 -- 172.21.15.123:0/969802481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8109560 msgr2=0x7fabb810b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 --2- 172.21.15.123:0/969802481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8109560 0x7fabb810b9f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009be0 tx=0x7fabb002bf80 comp rx=0 tx=0).stop 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 -- 172.21.15.123:0/969802481 shutdown_connections 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 --2- 172.21.15.123:0/969802481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8109560 0x7fabb810b9f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 --2- 172.21.15.123:0/969802481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8074080 0x7fabb80744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 --2- 172.21.15.123:0/969802481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8075660 0x7fabb8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 -- 172.21.15.123:0/969802481 >> 172.21.15.123:0/969802481 conn(0x7fabb80fcc40 msgr2=0x7fabb80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 -- 172.21.15.123:0/969802481 shutdown_connections 2024-03-24T00:41:29.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.676+0000 7fabbd800640 1 -- 172.21.15.123:0/969802481 wait complete. 2024-03-24T00:41:29.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.677+0000 7fabbd800640 1 Processor -- start 2024-03-24T00:41:29.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.677+0000 7fabbd800640 1 -- start start 2024-03-24T00:41:29.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8074080 0x7fabb8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 0x7fabb819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 0x7fabb81a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb810f590 con 0x7fabb8074080 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb810f410 con 0x7fabb8075660 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabbd800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb810f710 con 0x7fabb8109560 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 0x7fabb819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8074080 0x7fabb8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 0x7fabb81a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 0x7fabb819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56522/0 (socket says 172.21.15.123:56522) 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb6000640 1 -- 172.21.15.123:0/2373120818 learned_addr learned my addr 172.21.15.123:0/2373120818 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 -- 172.21.15.123:0/2373120818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 msgr2=0x7fabb819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 0x7fabb819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 -- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8074080 msgr2=0x7fabb8199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8074080 0x7fabb8199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.678+0000 7fabb7400640 1 -- 172.21.15.123:0/2373120818 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb81a17c0 con 0x7fabb8109560 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabb7400640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 0x7fabb81a1150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fabb0036040 tx=0x7fabb0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabaf400640 1 -- 172.21.15.123:0/2373120818 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0043020 con 0x7fabb8109560 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabaf400640 1 -- 172.21.15.123:0/2373120818 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabb00045f0 con 0x7fabb8109560 2024-03-24T00:41:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb81a1a50 con 0x7fabb8109560 2024-03-24T00:41:29.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabaf400640 1 -- 172.21.15.123:0/2373120818 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb003e7d0 con 0x7fabb8109560 2024-03-24T00:41:29.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.679+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb81a1ff0 con 0x7fabb8109560 2024-03-24T00:41:29.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.682+0000 7fabc05a8740 1 Processor -- start 2024-03-24T00:41:29.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.682+0000 7fabaf400640 1 -- 172.21.15.123:0/2373120818 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fabb0046020 con 0x7fabb8109560 2024-03-24T00:41:29.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 -- start start 2024-03-24T00:41:29.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 0x55ca053614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca05366c00 0x55ca05369090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca052c6eb0 con 0x55ca05366c00 2024-03-24T00:41:29.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca0526fa60 con 0x55ca053610a0 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabc05a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca052c2f50 con 0x55ca05361a90 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56870/0 (socket says 172.21.15.123:56870) 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabb6a00640 1 -- 172.21.15.123:0/1972271455 learned_addr learned my addr 172.21.15.123:0/1972271455 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabb6000640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 0x55ca053614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.683+0000 7fabb6000640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca05366c00 0x55ca05369090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 -- 172.21.15.123:0/1972271455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 msgr2=0x55ca053614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 0x55ca053614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 -- 172.21.15.123:0/1972271455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca05366c00 msgr2=0x55ca05369090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca05366c00 0x55ca05369090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 -- 172.21.15.123:0/1972271455 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca050a37e0 con 0x55ca05361a90 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6000640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 0x55ca053614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabaf400640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 0x7fab9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faba8009d20 tx=0x7faba8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0dbea2a489f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.684+0000 7fabaf400640 1 -- 172.21.15.123:0/2373120818 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fabb00f0570 con 0x7fabb8109560 2024-03-24T00:41:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7fabb6000640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 0x7fab9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7faba3e00640 1 -- 172.21.15.123:0/1972271455 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba801e070 con 0x55ca05361a90 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7faba3e00640 1 -- 172.21.15.123:0/1972271455 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba8019b10 con 0x55ca05361a90 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7faba3e00640 1 -- 172.21.15.123:0/1972271455 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba800ebf0 con 0x55ca05361a90 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7fabc05a8740 1 -- 172.21.15.123:0/1972271455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 msgr2=0x55ca05366590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7fabb6000640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 0x7fab9c0a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ca053622f0 tx=0x7faba4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.685+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faba8009d20 tx=0x7faba8011810 comp rx=0 tx=0).stop 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 -- 172.21.15.123:0/1972271455 shutdown_connections 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca05366c00 0x55ca05369090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05361a90 0x55ca05366590 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1972271455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca053610a0 0x55ca053614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 -- 172.21.15.123:0/1972271455 >> 172.21.15.123:0/1972271455 conn(0x55ca0535c610 msgr2=0x55ca0535ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 -- 172.21.15.123:0/1972271455 shutdown_connections 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 -- 172.21.15.123:0/1972271455 wait complete. 2024-03-24T00:41:29.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 -- start start 2024-03-24T00:41:29.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.686+0000 7fabc05a8740 1 Processor -- start 2024-03-24T00:41:29.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca05369bb0 con 0x55ca053610a0 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca05282eb0 con 0x55ca05361a90 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca05369d30 con 0x55ca05366c00 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabc05a8740 10 client.0 _ll_register_callbacks cb 0x55ca05277150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56530/0 (socket says 172.21.15.123:56530) 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 -- 172.21.15.123:0/1407221900 learned_addr learned my addr 172.21.15.123:0/1407221900 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb7400640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 -- 172.21.15.123:0/1407221900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 msgr2=0x55ca053728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 -- 172.21.15.123:0/1407221900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 msgr2=0x55ca0536b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.687+0000 7fabb6000640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7fabb6000640 1 -- 172.21.15.123:0/1407221900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca05103f70 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7fabb6000640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faba4003850 tx=0x7faba4009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d55e46b4eb85f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7faba0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba400fc00 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faba8019b10 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faba400fda0 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faba402a440 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7fabc05a8740 10 client.25880 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.688+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ca04f9b170 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba8019b10 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faba4028070 con 0x55ca05361a90 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 10 client.25880 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 10 client.25880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:29.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 20 client.25880 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca05377170 0x55ca05379610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 4 client.25880 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca052d43f0 con 0x55ca05377170 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.689+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca05377170 0x55ca05379610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.690+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca05377170 0x55ca05379610 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a736cdb1611d4ad server_cookie=58f12be39f2dbdbd in_seq=0 out_seq=0 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.690+0000 7faba0c00640 10 client.25880 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7faba0c00640 1 -- 172.21.15.123:0/1407221900 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ca052d43f0 con 0x55ca05377170 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7faba0c00640 10 client.25880 handle_command_reply: tid=0 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 client.25880 shutdown 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 msgr2=0x55ca0536b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faba4003850 tx=0x7faba4009800 comp rx=0 tx=0).stop 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 shutdown_connections 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca05377170 0x55ca05379610 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca0536b880 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1407221900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 >> 172.21.15.123:0/1407221900 conn(0x55ca0535c610 msgr2=0x55ca0535b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7faba0c00640 0 client.25880 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:29.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 shutdown_connections 2024-03-24T00:41:29.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 1 -- 172.21.15.123:0/1407221900 wait complete. 2024-03-24T00:41:29.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.691+0000 7fabc05a8740 20 client.25880 trim_cache size 0 max 16384 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.709+0000 7fabc05a8740 1 Processor -- start 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.709+0000 7fabc05a8740 1 -- start start 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.709+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0535ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca05362a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabc05a8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800c3c0 con 0x55ca053610a0 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabc05a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800e980 con 0x55ca05361a90 2024-03-24T00:41:29.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabc05a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800c9c0 con 0x55ca05366c00 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56538/0 (socket says 172.21.15.123:56538) 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 -- 172.21.15.123:0/3898385040 learned_addr learned my addr 172.21.15.123:0/3898385040 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6a00640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0535ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb7400640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca05362a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 -- 172.21.15.123:0/3898385040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 msgr2=0x55ca05362a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca05362a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 -- 172.21.15.123:0/3898385040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 msgr2=0x55ca0535ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0535ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 -- 172.21.15.123:0/3898385040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca05357c20 con 0x55ca05361a90 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb7400640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca05362a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:29.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.710+0000 7fabb6000640 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faba4010da0 tx=0x7faba4010dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b51fa2b48b6a2ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.711+0000 7faba3400640 1 -- 172.21.15.123:0/3898385040 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4020c10 con 0x55ca05361a90 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.711+0000 7faba3400640 1 -- 172.21.15.123:0/3898385040 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba4030030 con 0x55ca05361a90 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.711+0000 7faba3400640 1 -- 172.21.15.123:0/3898385040 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4004730 con 0x55ca05361a90 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.711+0000 7fabc05a8740 1 -- 172.21.15.123:0/3898385040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 msgr2=0x55ca05362530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.711+0000 7fabc05a8740 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faba4010da0 tx=0x7faba4010dd0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 -- 172.21.15.123:0/3898385040 shutdown_connections 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca05362a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca05362530 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 --2- 172.21.15.123:0/3898385040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0535ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 -- 172.21.15.123:0/3898385040 >> 172.21.15.123:0/3898385040 conn(0x55ca0535c610 msgr2=0x55ca0535f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 -- 172.21.15.123:0/3898385040 shutdown_connections 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 -- 172.21.15.123:0/3898385040 wait complete. 2024-03-24T00:41:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 -- start start 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.712+0000 7fabc05a8740 1 Processor -- start 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800d000 con 0x55ca053610a0 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800e320 con 0x55ca05361a90 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9800d660 con 0x55ca05366c00 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabc05a8740 10 client.0 _ll_register_callbacks cb 0x7faba4009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56554/0 (socket says 172.21.15.123:56554) 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33162/0 (socket says 172.21.15.123:33162) 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 -- 172.21.15.123:0/1141230998 learned_addr learned my addr 172.21.15.123:0/1141230998 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 -- 172.21.15.123:0/1141230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 msgr2=0x55ca053714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 -- 172.21.15.123:0/1141230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 msgr2=0x55ca0536ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.713+0000 7fabb6000640 1 -- 172.21.15.123:0/1141230998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca05371ac0 con 0x55ca05361a90 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7fabb6000640 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faba4020a20 tx=0x7faba40093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9743dbc873178829 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7faba0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:29.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4020c10 con 0x55ca05361a90 2024-03-24T00:41:29.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faba4030030 con 0x55ca05361a90 2024-03-24T00:41:29.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faba4005240 con 0x55ca05361a90 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faba4005440 con 0x55ca05361a90 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7fabc05a8740 10 client.25886 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.714+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ca053d3300 con 0x55ca05361a90 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba4030030 con 0x55ca05361a90 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faba4028040 con 0x55ca05361a90 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 10 client.25886 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 20 client.25886 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca053d3bc0 0x55ca053d5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 4 client.25886 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ca052ce740 con 0x55ca053d3bc0 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.715+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca053d3bc0 0x55ca053d5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.716+0000 7fabb6a00640 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca053d3bc0 0x55ca053d5fe0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f89fe4b3b048654 server_cookie=9583c8c7a7d88513 in_seq=0 out_seq=0 2024-03-24T00:41:29.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.716+0000 7faba0c00640 10 client.25886 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7faba0c00640 1 -- 172.21.15.123:0/1141230998 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ca052ce740 con 0x55ca053d3bc0 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7faba0c00640 10 client.25886 handle_command_reply: tid=0 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 client.25886 shutdown 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 msgr2=0x55ca053704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faba4020a20 tx=0x7faba40093d0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 shutdown_connections 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55ca053d3bc0 0x55ca053d5fe0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca05366c00 0x55ca053714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7faba0c00640 0 client.25886 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca05361a90 0x55ca053704c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 --2- 172.21.15.123:0/1141230998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ca053610a0 0x55ca0536ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 >> 172.21.15.123:0/1141230998 conn(0x55ca0535c610 msgr2=0x55ca05360100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 shutdown_connections 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 1 -- 172.21.15.123:0/1141230998 wait complete. 2024-03-24T00:41:29.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.717+0000 7fabc05a8740 20 client.25886 trim_cache size 0 max 16384 2024-03-24T00:41:29.722 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:41:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:41:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "64c90f45-07cd-46b8-893c-0ad586b4d093", 2024-03-24T00:41:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:41:29.722 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 msgr2=0x7fab9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 0x7fab9c0a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ca053622f0 tx=0x7faba4009260 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 msgr2=0x7fabb81a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 0x7fabb81a1150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fabb0036040 tx=0x7fabb0004270 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 shutdown_connections 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabb8109560 0x7fabb81a1150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb8075660 0x7fabb819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab9c0a2880 0x7fab9c0a4d40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 --2- 172.21.15.123:0/2373120818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabb8074080 0x7fabb8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 >> 172.21.15.123:0/2373120818 conn(0x7fabb80fcc40 msgr2=0x7fabb81054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 shutdown_connections 2024-03-24T00:41:29.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.718+0000 7fabbd800640 1 -- 172.21.15.123:0/2373120818 wait complete. 2024-03-24T00:41:29.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:41:29.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:41:29.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.982+0000 7f6822200640 1 -- 172.21.15.123:0/2482903635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 msgr2=0x7f681c100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.982+0000 7f6822200640 1 --2- 172.21.15.123:0/2482903635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c100cd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6808009900 tx=0x7f680802bc30 comp rx=0 tx=0).stop 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 -- 172.21.15.123:0/2482903635 shutdown_connections 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 --2- 172.21.15.123:0/2482903635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c101210 0x7f681c101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 --2- 172.21.15.123:0/2482903635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c100cd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 --2- 172.21.15.123:0/2482903635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c0fff10 0x7f681c100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 -- 172.21.15.123:0/2482903635 >> 172.21.15.123:0/2482903635 conn(0x7f681c0fbb30 msgr2=0x7f681c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 -- 172.21.15.123:0/2482903635 shutdown_connections 2024-03-24T00:41:29.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.983+0000 7f6822200640 1 -- 172.21.15.123:0/2482903635 wait complete. 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 Processor -- start 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 -- start start 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 0x7f681c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f550 con 0x7f681c100870 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f3d0 con 0x7f681c0fff10 2024-03-24T00:41:29.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f6822200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f6d0 con 0x7f681c101210 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f681b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f681b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56574/0 (socket says 172.21.15.123:56574) 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.984+0000 7f681b400640 1 -- 172.21.15.123:0/392945552 learned_addr learned my addr 172.21.15.123:0/392945552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681aa00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681be00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 0x7f681c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 -- 172.21.15.123:0/392945552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 msgr2=0x7f681c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 0x7f681c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 -- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 msgr2=0x7f681c19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 -- 172.21.15.123:0/392945552 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c1a1810 con 0x7f681c0fff10 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681aa00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681be00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 0x7f681c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f681b400640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f680c00e9f0 tx=0x7f680c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f6813e00640 1 -- 172.21.15.123:0/392945552 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c00cd80 con 0x7f681c0fff10 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f6813e00640 1 -- 172.21.15.123:0/392945552 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680c005500 con 0x7f681c0fff10 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c1a1b00 con 0x7f681c0fff10 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.985+0000 7f6813e00640 1 -- 172.21.15.123:0/392945552 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c010690 con 0x7f681c0fff10 2024-03-24T00:41:29.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.986+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1a1fd0 con 0x7f681c0fff10 2024-03-24T00:41:29.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.989+0000 7f682500a740 1 Processor -- start 2024-03-24T00:41:29.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.989+0000 7f682500a740 1 -- start start 2024-03-24T00:41:29.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.989+0000 7f6813e00640 1 -- 172.21.15.123:0/392945552 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f680c0046c0 con 0x7f681c0fff10 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.989+0000 7f682500a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.989+0000 7f682500a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0a9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f682500a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f682500a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40a009250 con 0x55b40a0a3e80 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f682500a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b409fb1a30 con 0x55b40a0a99e0 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f682500a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40a0052f0 con 0x55b40a0a4870 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56576/0 (socket says 172.21.15.123:56576) 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0a42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33178/0 (socket says 172.21.15.123:33178) 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 -- 172.21.15.123:0/3757342564 learned_addr learned my addr 172.21.15.123:0/3757342564 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681b400640 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 -- 172.21.15.123:0/3757342564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 msgr2=0x55b40a0a9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0a9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 -- 172.21.15.123:0/3757342564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 msgr2=0x55b40a0a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0a42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.990+0000 7f681aa00640 1 -- 172.21.15.123:0/3757342564 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b409f9c690 con 0x55b40a0a99e0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f680802c7b0 tx=0x7f68080095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc344bc4e9e7284f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f6807e00640 1 -- 172.21.15.123:0/3757342564 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808044020 con 0x55b40a0a99e0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f6807e00640 1 -- 172.21.15.123:0/3757342564 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680802cc40 con 0x55b40a0a99e0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f6807e00640 1 -- 172.21.15.123:0/3757342564 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808004920 con 0x55b40a0a99e0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f6813e00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 0x7f68000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.991+0000 7f6813e00640 1 -- 172.21.15.123:0/392945552 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f680c0cf070 con 0x7f681c0fff10 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f681be00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 0x7f68000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 -- 172.21.15.123:0/3757342564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 msgr2=0x55b40a0abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f680802c7b0 tx=0x7f68080095e0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 -- 172.21.15.123:0/3757342564 shutdown_connections 2024-03-24T00:41:29.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0abe70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0a9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 --2- 172.21.15.123:0/3757342564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 -- 172.21.15.123:0/3757342564 >> 172.21.15.123:0/3757342564 conn(0x55b40a06c200 msgr2=0x55b40a09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.992+0000 7f682500a740 1 -- 172.21.15.123:0/3757342564 shutdown_connections 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.993+0000 7f681be00640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 0x7f68000a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55b40a0a50d0 tx=0x7f68140098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.993+0000 7f682500a740 1 -- 172.21.15.123:0/3757342564 wait complete. 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.993+0000 7f682500a740 1 -- start start 2024-03-24T00:41:29.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.993+0000 7f682500a740 1 Processor -- start 2024-03-24T00:41:29.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.993+0000 7f682500a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a99e0 0x55b40a0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40a0ac930 con 0x55b40a0a99e0 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b409fc4e80 con 0x55b40a0a4870 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40a0acab0 con 0x55b40a0a3e80 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f682500a740 10 client.0 _ll_register_callbacks cb 0x55b409fd7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56930/0 (socket says 172.21.15.123:56930) 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56586/0 (socket says 172.21.15.123:56586) 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681b400640 1 -- 172.21.15.123:0/2913434618 learned_addr learned my addr 172.21.15.123:0/2913434618 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681be00640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a99e0 0x55b40a0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 -- 172.21.15.123:0/2913434618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 msgr2=0x55b40a0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 -- 172.21.15.123:0/2913434618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a99e0 msgr2=0x55b40a0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a99e0 0x55b40a0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681aa00640 1 -- 172.21.15.123:0/2913434618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b409de77d0 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.994+0000 7f681b400640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f681aa00640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f680802c240 tx=0x7f680802cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fbc76167ce8fe64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f6804c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680804f070 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f68080055f0 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f680802cc40 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6808044040 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f682500a740 10 client.25898 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b409d1d3f0 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.995+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680802cc40 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6808034600 con 0x55b40a0a4870 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 10 client.25898 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 10 client.25898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:30.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 20 client.25898 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a0b97e0 0x55b40a0bbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 4 client.25898 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b40a016790 con 0x55b40a0b97e0 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f681b400640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a0b97e0 0x55b40a0bbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f681b400640 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a0b97e0 0x55b40a0bbbd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25c57e027a925e44 server_cookie=cbf82988f3007e50 in_seq=0 out_seq=0 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.996+0000 7f6804c00640 10 client.25898 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f6804c00640 1 -- 172.21.15.123:0/2913434618 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b40a016790 con 0x55b40a0b97e0 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f6804c00640 10 client.25898 handle_command_reply: tid=0 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 client.25898 shutdown 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 msgr2=0x55b40a0ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f680802c240 tx=0x7f680802cdf0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 shutdown_connections 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a99e0 0x55b40a0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0ae2b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a0b97e0 0x55b40a0bbbd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 --2- 172.21.15.123:0/2913434618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a3e80 0x55b40a0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 >> 172.21.15.123:0/2913434618 conn(0x55b40a06c200 msgr2=0x55b40a09ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.997+0000 7f6804c00640 0 client.25898 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.998+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 shutdown_connections 2024-03-24T00:41:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.998+0000 7f682500a740 1 -- 172.21.15.123:0/2913434618 wait complete. 2024-03-24T00:41:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:29.998+0000 7f682500a740 20 client.25898 trim_cache size 0 max 16384 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.016+0000 7f682500a740 1 Processor -- start 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.016+0000 7f682500a740 1 -- start start 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a09e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a99e0 0x55b40a0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a8c30 con 0x55b40a0a3e80 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1ab1f0 con 0x55b40a0a4870 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f682500a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a9230 con 0x55b40a0a99e0 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a09e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56596/0 (socket says 172.21.15.123:56596) 2024-03-24T00:41:30.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a99e0 0x55b40a0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 -- 172.21.15.123:0/2492776882 learned_addr learned my addr 172.21.15.123:0/2492776882 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 -- 172.21.15.123:0/2492776882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a99e0 msgr2=0x55b40a0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a99e0 0x55b40a0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 -- 172.21.15.123:0/2492776882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 msgr2=0x55b40a09e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a09e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.017+0000 7f681aa00640 1 -- 172.21.15.123:0/2492776882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b409cdd0f0 con 0x55b40a0a4870 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f681aa00640 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f68080349f0 tx=0x7f6808009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=341494ed962a12e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f6807400640 1 -- 172.21.15.123:0/2492776882 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680804f040 con 0x55b40a0a4870 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f6807400640 1 -- 172.21.15.123:0/2492776882 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6808040e40 con 0x55b40a0a4870 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f6807400640 1 -- 172.21.15.123:0/2492776882 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808005400 con 0x55b40a0a4870 2024-03-24T00:41:30.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f682500a740 1 -- 172.21.15.123:0/2492776882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 msgr2=0x55b40a0b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.018+0000 7f682500a740 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f68080349f0 tx=0x7f6808009780 comp rx=0 tx=0).stop 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 -- 172.21.15.123:0/2492776882 shutdown_connections 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a99e0 0x55b40a0a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a4870 0x55b40a0b52b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 --2- 172.21.15.123:0/2492776882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a09e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 -- 172.21.15.123:0/2492776882 >> 172.21.15.123:0/2492776882 conn(0x55b40a06c200 msgr2=0x55b40a09e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 -- 172.21.15.123:0/2492776882 shutdown_connections 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 -- 172.21.15.123:0/2492776882 wait complete. 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 -- start start 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.019+0000 7f682500a740 1 Processor -- start 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a9870 con 0x55b40a0a3e80 2024-03-24T00:41:30.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1aab90 con 0x55b40a0a99e0 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a9ed0 con 0x55b40a0a4870 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f682500a740 10 client.0 _ll_register_callbacks cb 0x7f6814009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56956/0 (socket says 172.21.15.123:56956) 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 -- 172.21.15.123:0/3757796800 learned_addr learned my addr 172.21.15.123:0/3757796800 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:30.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 -- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 msgr2=0x55b40a0b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 -- 172.21.15.123:0/3757796800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 msgr2=0x55b40a0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681aa00640 1 -- 172.21.15.123:0/3757796800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b40a0b4b90 con 0x55b40a0a4870 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.020+0000 7f681b400640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.021+0000 7f681aa00640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6808034a70 tx=0x7f6808034da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fff005d7ce7bf10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.021+0000 7f6804c00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.021+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808005140 con 0x55b40a0a4870 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.021+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f68080052e0 con 0x55b40a0a4870 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.021+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b40a010ae0 con 0x55b40a0a4870 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6808005520 con 0x55b40a0a4870 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 10 client.25906 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:30.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b40a115560 con 0x55b40a0a4870 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b40a010ae0 con 0x55b40a0a4870 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f68080684e0 con 0x55b40a0a4870 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 10 client.25906 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 20 client.25906 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a115d70 0x55b40a118190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 4 client.25906 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b409f8e3a0 con 0x55b40a115d70 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.022+0000 7f681b400640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a115d70 0x55b40a118190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f6804c00640 10 client.25906 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f681b400640 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a115d70 0x55b40a118190 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6485ac3875c66130 server_cookie=1292ab9b6c368227 in_seq=0 out_seq=0 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f6804c00640 1 -- 172.21.15.123:0/3757796800 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55b409f8e3a0 con 0x55b40a115d70 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f6804c00640 10 client.25906 handle_command_reply: tid=0 2024-03-24T00:41:30.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 client.25906 shutdown 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 msgr2=0x55b40a0b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6808034a70 tx=0x7f6808034da0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 shutdown_connections 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b40a0a99e0 0x55b40a0b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b40a115d70 0x55b40a118190 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b40a0a4870 0x55b40a0b3590 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 --2- 172.21.15.123:0/3757796800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b40a0a3e80 0x55b40a0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f6804c00640 0 client.25906 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 >> 172.21.15.123:0/3757796800 conn(0x55b40a06c200 msgr2=0x55b40a0a32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 shutdown_connections 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.023+0000 7f682500a740 1 -- 172.21.15.123:0/3757796800 wait complete. 2024-03-24T00:41:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f682500a740 20 client.25906 trim_cache size 0 max 16384 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "64c90f45-07cd-46b8-893c-0ad586b4d093": { 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "64c90f45-07cd-46b8-893c-0ad586b4d093", 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:41:30.029 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 msgr2=0x7f68000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 0x7f68000a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55b40a0a50d0 tx=0x7f68140098a0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 msgr2=0x7f681c199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f680c00e9f0 tx=0x7f680c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 shutdown_connections 2024-03-24T00:41:30.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f681c101210 0x7f681c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f68000a2800 0x7f68000a4cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c100870 0x7f681c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 --2- 172.21.15.123:0/392945552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f681c0fff10 0x7f681c199bf0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 >> 172.21.15.123:0/392945552 conn(0x7f681c0fbb30 msgr2=0x7f681c0fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 shutdown_connections 2024-03-24T00:41:30.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:41:30.024+0000 7f6822200640 1 -- 172.21.15.123:0/392945552 wait complete. 2024-03-24T00:41:30.042 INFO:tasks.cephfs.filesystem:scrub status for tag:64c90f45-07cd-46b8-893c-0ad586b4d093 - {'path': '/', 'tag': '64c90f45-07cd-46b8-893c-0ad586b4d093', 'options': 'recursive,force'} 2024-03-24T00:41:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:30 smithi204 ceph-mon[36345]: pgmap v493: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 58 MiB/s wr, 6.54k op/s 2024-03-24T00:41:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:30 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:41:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:30 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:30.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:30 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:41:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:30 smithi123 ceph-mon[31487]: pgmap v493: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 58 MiB/s wr, 6.54k op/s 2024-03-24T00:41:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:30 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:41:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:30 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:30.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:30 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:41:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:30 smithi177 ceph-mon[36034]: pgmap v493: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 58 MiB/s wr, 6.54k op/s 2024-03-24T00:41:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:30 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:41:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:30 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:30.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:30 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:41:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.933+0000 7f6d1f400640 1 -- 172.21.15.123:0/843254656 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18032340 con 0x7f6d20103280 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 -- 172.21.15.123:0/843254656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 msgr2=0x7f6d20109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 --2- 172.21.15.123:0/843254656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d20109b10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009860 tx=0x7f6d1802bb70 comp rx=0 tx=0).stop 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 -- 172.21.15.123:0/843254656 shutdown_connections 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 --2- 172.21.15.123:0/843254656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d20109b10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 --2- 172.21.15.123:0/843254656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d20102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 --2- 172.21.15.123:0/843254656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 0x7f6d20101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 -- 172.21.15.123:0/843254656 >> 172.21.15.123:0/843254656 conn(0x7f6d200fcc60 msgr2=0x7f6d200ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 -- 172.21.15.123:0/843254656 shutdown_connections 2024-03-24T00:41:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.934+0000 7f6d26e00640 1 -- 172.21.15.123:0/843254656 wait complete. 2024-03-24T00:41:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.935+0000 7f6d26e00640 1 Processor -- start 2024-03-24T00:41:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 -- start start 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 0x7f6d20199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d201a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20075120 con 0x7f6d201016d0 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20074fa0 con 0x7f6d201028c0 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d26e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d200752a0 con 0x7f6d20103280 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d201a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d201a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56984/0 (socket says 172.21.15.123:56984) 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d24e00640 1 -- 172.21.15.123:0/1610837479 learned_addr learned my addr 172.21.15.123:0/1610837479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 0x7f6d20199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.936+0000 7f6d17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56624/0 (socket says 172.21.15.123:56624) 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 -- 172.21.15.123:0/1610837479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 msgr2=0x7f6d201a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d201a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 -- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 msgr2=0x7f6d20199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 0x7f6d20199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 -- 172.21.15.123:0/1610837479 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d201a1700 con 0x7f6d201028c0 2024-03-24T00:41:31.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.937+0000 7f6d17400640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e9f0 tx=0x7f6d0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.938+0000 7f6d1d600640 1 -- 172.21.15.123:0/1610837479 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0800ccc0 con 0x7f6d201028c0 2024-03-24T00:41:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.938+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d201a19f0 con 0x7f6d201028c0 2024-03-24T00:41:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.938+0000 7f6d1d600640 1 -- 172.21.15.123:0/1610837479 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d08005020 con 0x7f6d201028c0 2024-03-24T00:41:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.938+0000 7f6d1d600640 1 -- 172.21.15.123:0/1610837479 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d080049e0 con 0x7f6d201028c0 2024-03-24T00:41:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.938+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d201a1f40 con 0x7f6d201028c0 2024-03-24T00:41:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 Processor -- start 2024-03-24T00:41:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d1d600640 1 -- 172.21.15.123:0/1610837479 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6d08005560 con 0x7f6d201028c0 2024-03-24T00:41:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 -- start start 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e892b23b0 con 0x562e892ef1e0 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e89254830 con 0x562e892f4d40 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.943+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e891fd5d0 con 0x562e892efbd0 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56986/0 (socket says 172.21.15.123:56986) 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 -- 172.21.15.123:0/143754370 learned_addr learned my addr 172.21.15.123:0/143754370 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d17400640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d24e00640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 -- 172.21.15.123:0/143754370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 msgr2=0x562e892f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 -- 172.21.15.123:0/143754370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 msgr2=0x562e892ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 -- 172.21.15.123:0/143754370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e892152c0 con 0x562e892efbd0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d24e00640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d17400640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.944+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e480 tx=0x7f6d100095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c466c3575bb6917 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.945+0000 7f6d15600640 1 -- 172.21.15.123:0/143754370 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10004b20 con 0x562e892efbd0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.945+0000 7f6d15600640 1 -- 172.21.15.123:0/143754370 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d1001e070 con 0x562e892efbd0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.945+0000 7f6d15600640 1 -- 172.21.15.123:0/143754370 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100213e0 con 0x562e892efbd0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.945+0000 7f6d1d600640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 0x7f6d040a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.945+0000 7f6d1d600640 1 -- 172.21.15.123:0/1610837479 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6d080cea60 con 0x7f6d201028c0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 0x7f6d040a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 -- 172.21.15.123:0/143754370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 msgr2=0x562e892f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e480 tx=0x7f6d100095c0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.073 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 -- 172.21.15.123:0/143754370 shutdown_connections 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e892f46d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/143754370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.946+0000 7f6d29a7c740 1 -- 172.21.15.123:0/143754370 >> 172.21.15.123:0/143754370 conn(0x562e892ea750 msgr2=0x562e892eab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.947+0000 7f6d29a7c740 1 -- 172.21.15.123:0/143754370 shutdown_connections 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.947+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 0x7f6d040a4d60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x562e892f0430 tx=0x7f6d180058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.947+0000 7f6d29a7c740 1 -- 172.21.15.123:0/143754370 wait complete. 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.948+0000 7f6d29a7c740 1 -- start start 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.948+0000 7f6d29a7c740 1 Processor -- start 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.948+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 0x562e892f9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e89300950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e89210a20 con 0x562e892ef1e0 2024-03-24T00:41:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e892508d0 con 0x562e892efbd0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e892f7cf0 con 0x562e892f4d40 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d29a7c740 10 client.0 _ll_register_callbacks cb 0x562e8924d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 0x562e892f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 0x562e892f9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56650/0 (socket says 172.21.15.123:56650) 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d17400640 1 -- 172.21.15.123:0/1666279130 learned_addr learned my addr 172.21.15.123:0/1666279130 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e89300950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.949+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 -- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 msgr2=0x562e892f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 0x562e892f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 -- 172.21.15.123:0/1666279130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 msgr2=0x562e892f93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 -- 172.21.15.123:0/1666279130 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e88f69210 con 0x562e892f4d40 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d24e00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e89300950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d180057c0 tx=0x7f6d18005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c770a9cccd7817ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d0e000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.950+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1804e020 con 0x562e892f4d40 2024-03-24T00:41:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6d1802c510 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d1001e070 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d18039040 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d29a7c740 10 client.25918 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562e89035620 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.951+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d1001e070 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6d180035f0 con 0x562e892f4d40 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 10 client.25918 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 20 client.25918 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89307600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 4 client.25918 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e89261d70 con 0x562e89305210 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.952+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89307600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.953+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89307600 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ba49592361d91db server_cookie=c42ceeb0b324aa3c in_seq=0 out_seq=0 2024-03-24T00:41:32.076 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.953+0000 7f6d0e000640 10 client.25918 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.953+0000 7f6d0e000640 1 -- 172.21.15.123:0/1666279130 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562e89261d70 con 0x562e89305210 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.953+0000 7f6d0e000640 10 client.25918 handle_command_reply: tid=0 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 client.25918 shutdown 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 msgr2=0x562e89300950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e89300950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d180057c0 tx=0x7f6d18005f90 comp rx=0 tx=0).stop 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 shutdown_connections 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89307600 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e89300950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d0e000640 0 client.25918 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892efbd0 0x562e892f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/1666279130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 >> 172.21.15.123:0/1666279130 conn(0x562e892ea750 msgr2=0x562e892e9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 shutdown_connections 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.954+0000 7f6d29a7c740 1 -- 172.21.15.123:0/1666279130 wait complete. 2024-03-24T00:41:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.955+0000 7f6d29a7c740 20 client.25918 trim_cache size 0 max 16384 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.974+0000 7f6d29a7c740 1 Processor -- start 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 -- start start 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a47b0 con 0x562e892ef1e0 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a6d70 con 0x562e892f4d40 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a4db0 con 0x562e892efbd0 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57018/0 (socket says 172.21.15.123:57018) 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 -- 172.21.15.123:0/3390556951 learned_addr learned my addr 172.21.15.123:0/3390556951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d24e00640 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 -- 172.21.15.123:0/3390556951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 msgr2=0x562e892f0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 -- 172.21.15.123:0/3390556951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 msgr2=0x562e892f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.975+0000 7f6d17400640 1 -- 172.21.15.123:0/3390556951 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e892f7e30 con 0x562e892efbd0 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.976+0000 7f6d17400640 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c6a0 tx=0x7f6d080054d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49fe516c872f615c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.976+0000 7f6d14c00640 1 -- 172.21.15.123:0/3390556951 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0801c030 con 0x562e892efbd0 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.976+0000 7f6d14c00640 1 -- 172.21.15.123:0/3390556951 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d080cee50 con 0x562e892efbd0 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.976+0000 7f6d14c00640 1 -- 172.21.15.123:0/3390556951 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08097a20 con 0x562e892efbd0 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- 172.21.15.123:0/3390556951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 msgr2=0x562e89300910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c6a0 tx=0x7f6d080054d0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- 172.21.15.123:0/3390556951 shutdown_connections 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892f4d40 0x562e892f0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892efbd0 0x562e89300910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/3390556951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892ef1e0 0x562e892f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- 172.21.15.123:0/3390556951 >> 172.21.15.123:0/3390556951 conn(0x562e892ea750 msgr2=0x562e892eab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- 172.21.15.123:0/3390556951 shutdown_connections 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- 172.21.15.123:0/3390556951 wait complete. 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 -- start start 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.977+0000 7f6d29a7c740 1 Processor -- start 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892efbd0 0x562e892fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e892ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a53f0 con 0x562e892efbd0 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a6710 con 0x562e892ef1e0 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d201a5a50 con 0x562e892f4d40 2024-03-24T00:41:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d29a7c740 10 client.0 _ll_register_callbacks cb 0x7f6d10009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e892ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:56666/0 (socket says 172.21.15.123:56666) 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d1fe00640 1 -- 172.21.15.123:0/737718501 learned_addr learned my addr 172.21.15.123:0/737718501 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e892ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57026/0 (socket says 172.21.15.123:57026) 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.978+0000 7f6d17400640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892efbd0 0x562e892fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 -- 172.21.15.123:0/737718501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 msgr2=0x562e892ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e892ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 -- 172.21.15.123:0/737718501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892efbd0 msgr2=0x562e892fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892efbd0 0x562e892fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 -- 172.21.15.123:0/737718501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e892ffff0 con 0x562e892ef1e0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d1fe00640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e450 tx=0x7f6d10005f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db7384800c353e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d0e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1001a670 con 0x562e892ef1e0 2024-03-24T00:41:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6d1002c4b0 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d080cee50 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d1002c020 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d29a7c740 10 client.25922 fetch_fsmap learned FSMap version 19 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.979+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562e89360f30 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d080cee50 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6d1000b040 con 0x562e892ef1e0 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 10 client.25922 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 10 client.25922 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 20 client.25922 populate_metadata read hostname 'smithi123' 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89363750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 4 client.25922 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.980+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562e8925c0c0 con 0x562e89305210 2024-03-24T00:41:32.082 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.981+0000 7f6d17400640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89363750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.083 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.981+0000 7f6d0e000640 10 client.25922 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:32.083 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:31.981+0000 7f6d17400640 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89363750 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4457397c63d4a1b server_cookie=8271fe0489f20c26 in_seq=0 out_seq=0 2024-03-24T00:41:32.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:32 smithi204 ceph-mon[36345]: pgmap v494: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 64 MiB/s wr, 7.10k op/s 2024-03-24T00:41:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:32 smithi123 ceph-mon[31487]: pgmap v494: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 64 MiB/s wr, 7.10k op/s 2024-03-24T00:41:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:32 smithi177 ceph-mon[36034]: pgmap v494: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 64 MiB/s wr, 7.10k op/s 2024-03-24T00:41:34.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:34 smithi204 ceph-mon[36345]: pgmap v495: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 59 MiB/s wr, 6.51k op/s 2024-03-24T00:41:34.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:34 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:34 smithi123 ceph-mon[31487]: pgmap v495: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 59 MiB/s wr, 6.51k op/s 2024-03-24T00:41:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:34 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:34 smithi177 ceph-mon[36034]: pgmap v495: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 59 MiB/s wr, 6.51k op/s 2024-03-24T00:41:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:34 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:41:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:36 smithi204 ceph-mon[36345]: pgmap v496: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 62 MiB/s wr, 7.03k op/s 2024-03-24T00:41:36.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:36 smithi123 ceph-mon[31487]: pgmap v496: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 62 MiB/s wr, 7.03k op/s 2024-03-24T00:41:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:36 smithi177 ceph-mon[36034]: pgmap v496: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 62 MiB/s wr, 7.03k op/s 2024-03-24T00:41:37.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:37 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:41:37.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:37 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:41:37.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:37 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:41:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:38 smithi123 ceph-mon[31487]: pgmap v497: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 54 MiB/s wr, 6.23k op/s 2024-03-24T00:41:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:38 smithi177 ceph-mon[36034]: pgmap v497: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 54 MiB/s wr, 6.23k op/s 2024-03-24T00:41:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:38 smithi204 ceph-mon[36345]: pgmap v497: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 54 MiB/s wr, 6.23k op/s 2024-03-24T00:41:40.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:40 smithi123 ceph-mon[31487]: pgmap v498: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 59 MiB/s wr, 6.93k op/s 2024-03-24T00:41:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:40 smithi177 ceph-mon[36034]: pgmap v498: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 59 MiB/s wr, 6.93k op/s 2024-03-24T00:41:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:40 smithi204 ceph-mon[36345]: pgmap v498: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 59 MiB/s wr, 6.93k op/s 2024-03-24T00:41:42.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:42 smithi123 ceph-mon[31487]: pgmap v499: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 64 MiB/s wr, 7.36k op/s 2024-03-24T00:41:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:42 smithi177 ceph-mon[36034]: pgmap v499: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 64 MiB/s wr, 7.36k op/s 2024-03-24T00:41:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:42 smithi204 ceph-mon[36345]: pgmap v499: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 64 MiB/s wr, 7.36k op/s 2024-03-24T00:41:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:44 smithi123 ceph-mon[31487]: pgmap v500: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 56 MiB/s wr, 6.63k op/s 2024-03-24T00:41:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:44 smithi177 ceph-mon[36034]: pgmap v500: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 56 MiB/s wr, 6.63k op/s 2024-03-24T00:41:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:44 smithi204 ceph-mon[36345]: pgmap v500: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 56 MiB/s wr, 6.63k op/s 2024-03-24T00:41:46.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:46 smithi123 ceph-mon[31487]: pgmap v501: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 60 MiB/s wr, 6.88k op/s 2024-03-24T00:41:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:46 smithi177 ceph-mon[36034]: pgmap v501: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 60 MiB/s wr, 6.88k op/s 2024-03-24T00:41:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:46 smithi204 ceph-mon[36345]: pgmap v501: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 60 MiB/s wr, 6.88k op/s 2024-03-24T00:41:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:48 smithi177 ceph-mon[36034]: pgmap v502: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 53 MiB/s wr, 5.95k op/s 2024-03-24T00:41:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.774+0000 7f6d0e000640 1 -- 172.21.15.123:0/737718501 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562e8925c0c0 con 0x562e89305210 2024-03-24T00:41:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.774+0000 7f6d0e000640 10 client.25922 handle_command_reply: tid=0 2024-03-24T00:41:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 client.25922 shutdown 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 msgr2=0x562e892fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e450 tx=0x7f6d10005f40 comp rx=0 tx=0).stop 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 shutdown_connections 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562e89305210 0x562e89363750 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d0e000640 0 client.25922 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.775+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e892f4d40 0x562e892ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562e892efbd0 0x562e892fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 1 --2- 172.21.15.123:0/737718501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e892ef1e0 0x562e892fe4b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 >> 172.21.15.123:0/737718501 conn(0x562e892ea750 msgr2=0x562e892f0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 shutdown_connections 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 1 -- 172.21.15.123:0/737718501 wait complete. 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.776+0000 7f6d29a7c740 20 client.25922 trim_cache size 0 max 16384 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:41:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:41:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 msgr2=0x7f6d040a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 0x7f6d040a4d60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x562e892f0430 tx=0x7f6d180058b0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 msgr2=0x7f6d2019a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e9f0 tx=0x7f6d0800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 shutdown_connections 2024-03-24T00:41:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d20103280 0x7f6d201a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d201028c0 0x7f6d2019a020 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6d040a28a0 0x7f6d040a4d60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 --2- 172.21.15.123:0/1610837479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d201016d0 0x7f6d20199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 >> 172.21.15.123:0/1610837479 conn(0x7f6d200fcc60 msgr2=0x7f6d20105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.779+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 shutdown_connections 2024-03-24T00:41:48.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:41:48.780+0000 7f6d26e00640 1 -- 172.21.15.123:0/1610837479 wait complete. 2024-03-24T00:41:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:48 smithi204 ceph-mon[36345]: pgmap v502: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 53 MiB/s wr, 5.95k op/s 2024-03-24T00:41:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:48 smithi123 ceph-mon[31487]: pgmap v502: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 53 MiB/s wr, 5.95k op/s 2024-03-24T00:41:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:50 smithi177 ceph-mon[36034]: pgmap v503: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 65 MiB/s wr, 7.15k op/s 2024-03-24T00:41:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:50 smithi204 ceph-mon[36345]: pgmap v503: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 65 MiB/s wr, 7.15k op/s 2024-03-24T00:41:50.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:50 smithi123 ceph-mon[31487]: pgmap v503: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 65 MiB/s wr, 7.15k op/s 2024-03-24T00:41:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:52 smithi177 ceph-mon[36034]: pgmap v504: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 70 MiB/s wr, 7.50k op/s 2024-03-24T00:41:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:52 smithi204 ceph-mon[36345]: pgmap v504: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 70 MiB/s wr, 7.50k op/s 2024-03-24T00:41:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:52 smithi123 ceph-mon[31487]: pgmap v504: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 70 MiB/s wr, 7.50k op/s 2024-03-24T00:41:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:54 smithi177 ceph-mon[36034]: pgmap v505: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 65 MiB/s wr, 6.85k op/s 2024-03-24T00:41:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:54 smithi204 ceph-mon[36345]: pgmap v505: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 65 MiB/s wr, 6.85k op/s 2024-03-24T00:41:54.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:54 smithi123 ceph-mon[31487]: pgmap v505: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 65 MiB/s wr, 6.85k op/s 2024-03-24T00:41:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:56 smithi177 ceph-mon[36034]: pgmap v506: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 67 MiB/s wr, 6.96k op/s 2024-03-24T00:41:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:56 smithi204 ceph-mon[36345]: pgmap v506: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 67 MiB/s wr, 6.96k op/s 2024-03-24T00:41:56.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:56 smithi123 ceph-mon[31487]: pgmap v506: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 67 MiB/s wr, 6.96k op/s 2024-03-24T00:41:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:41:58 smithi177 ceph-mon[36034]: pgmap v507: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 59 MiB/s wr, 6.05k op/s 2024-03-24T00:41:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:41:58 smithi204 ceph-mon[36345]: pgmap v507: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 59 MiB/s wr, 6.05k op/s 2024-03-24T00:41:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:41:58 smithi123 ceph-mon[31487]: pgmap v507: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 59 MiB/s wr, 6.05k op/s 2024-03-24T00:42:00.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:42:00.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 -- 172.21.15.123:0/2317189132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c103150 msgr2=0x7fda3c1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 --2- 172.21.15.123:0/2317189132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c103150 0x7fda3c1099e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fda34009900 tx=0x7fda3402bc50 comp rx=0 tx=0).stop 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 -- 172.21.15.123:0/2317189132 shutdown_connections 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 --2- 172.21.15.123:0/2317189132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c103150 0x7fda3c1099e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 --2- 172.21.15.123:0/2317189132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c102790 0x7fda3c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 --2- 172.21.15.123:0/2317189132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 0x7fda3c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 -- 172.21.15.123:0/2317189132 >> 172.21.15.123:0/2317189132 conn(0x7fda3c0fcb10 msgr2=0x7fda3c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 -- 172.21.15.123:0/2317189132 shutdown_connections 2024-03-24T00:42:00.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.252+0000 7fda43000640 1 -- 172.21.15.123:0/2317189132 wait complete. 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 Processor -- start 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 -- start start 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 0x7fda3c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 0x7fda3c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c10d270 con 0x7fda3c1015a0 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c10d0f0 con 0x7fda3c102790 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.253+0000 7fda43000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c10d3f0 con 0x7fda3c103150 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 0x7fda3c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47572/0 (socket says 172.21.15.123:47572) 2024-03-24T00:42:00.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 -- 172.21.15.123:0/344625471 learned_addr learned my addr 172.21.15.123:0/344625471 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda42600640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 0x7fda3c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 -- 172.21.15.123:0/344625471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 msgr2=0x7fda3c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 0x7fda3c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 -- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 msgr2=0x7fda3c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 0x7fda3c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 -- 172.21.15.123:0/344625471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c074860 con 0x7fda3c102790 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda42600640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 0x7fda3c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.254+0000 7fda41200640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e9f0 tx=0x7fda2800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.255+0000 7fda3a000640 1 -- 172.21.15.123:0/344625471 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2800cd80 con 0x7fda3c102790 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.255+0000 7fda3a000640 1 -- 172.21.15.123:0/344625471 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda28005500 con 0x7fda3c102790 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.255+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c1a1950 con 0x7fda3c102790 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.255+0000 7fda3a000640 1 -- 172.21.15.123:0/344625471 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28010690 con 0x7fda3c102790 2024-03-24T00:42:00.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.255+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1a1e50 con 0x7fda3c102790 2024-03-24T00:42:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.258+0000 7fda4738c740 1 Processor -- start 2024-03-24T00:42:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 -- start start 2024-03-24T00:42:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda3a000640 1 -- 172.21.15.123:0/344625471 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fda280046c0 con 0x7fda3c102790 2024-03-24T00:42:00.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 0x55eeb33a8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 0x55eeb33b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb330d250 con 0x55eeb33a8050 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb32b5a30 con 0x55eeb33a8a40 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda4738c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb33092f0 con 0x55eeb33adbb0 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda42600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 0x55eeb33a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda41200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 0x55eeb33b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda42600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 0x55eeb33a8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55248/0 (socket says 172.21.15.123:55248) 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda42600640 1 -- 172.21.15.123:0/2501879231 learned_addr learned my addr 172.21.15.123:0/2501879231 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.259+0000 7fda41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47576/0 (socket says 172.21.15.123:47576) 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 -- 172.21.15.123:0/2501879231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 msgr2=0x55eeb33ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 -- 172.21.15.123:0/2501879231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 msgr2=0x55eeb33a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 0x55eeb33a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 -- 172.21.15.123:0/2501879231 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeb32a0690 con 0x55eeb33adbb0 2024-03-24T00:42:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.260+0000 7fda41200640 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 0x55eeb33b0040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda280cca30 tx=0x7fda280d2f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc568bc33ac3203 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda32a00640 1 -- 172.21.15.123:0/2501879231 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280e2070 con 0x55eeb33adbb0 2024-03-24T00:42:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda3a000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 0x7fda200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda32a00640 1 -- 172.21.15.123:0/2501879231 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda280d7c80 con 0x55eeb33adbb0 2024-03-24T00:42:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda32a00640 1 -- 172.21.15.123:0/2501879231 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280cf630 con 0x55eeb33adbb0 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda3a000640 1 -- 172.21.15.123:0/344625471 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fda280cefe0 con 0x7fda3c102790 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.261+0000 7fda42600640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 0x7fda200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- 172.21.15.123:0/2501879231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 msgr2=0x55eeb33b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 0x55eeb33b0040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda280cca30 tx=0x7fda280d2f80 comp rx=0 tx=0).stop 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- 172.21.15.123:0/2501879231 shutdown_connections 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33adbb0 0x55eeb33b0040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 --2- 172.21.15.123:0/2501879231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33a8050 0x55eeb33a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- 172.21.15.123:0/2501879231 >> 172.21.15.123:0/2501879231 conn(0x55eeb3370200 msgr2=0x55eeb33a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda42600640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 0x7fda200a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda34005c40 tx=0x7fda3402c690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- 172.21.15.123:0/2501879231 shutdown_connections 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- 172.21.15.123:0/2501879231 wait complete. 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.262+0000 7fda4738c740 1 -- start start 2024-03-24T00:42:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 Processor -- start 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 0x55eeb33b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb33b0b60 con 0x55eeb33adbb0 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb32c8e80 con 0x55eeb33a8050 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeb33b0ce0 con 0x55eeb33a8a40 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda4738c740 10 client.0 _ll_register_callbacks cb 0x55eeb32db130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.263+0000 7fda41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda42600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 0x55eeb33b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47580/0 (socket says 172.21.15.123:47580) 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 -- 172.21.15.123:0/4030319450 learned_addr learned my addr 172.21.15.123:0/4030319450 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 -- 172.21.15.123:0/4030319450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 msgr2=0x55eeb33b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 0x55eeb33b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 -- 172.21.15.123:0/4030319450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 msgr2=0x55eeb33b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41c00640 1 -- 172.21.15.123:0/4030319450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eeb30eb7d0 con 0x55eeb33a8050 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda41200640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 0x55eeb33b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.264+0000 7fda42600640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda41c00640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fda2c012cf0 tx=0x7fda2c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb722beb9b8680 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda27400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c012110 con 0x55eeb33a8050 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fda2c003970 con 0x55eeb33a8050 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda280d7c80 con 0x55eeb33a8050 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda2c0122b0 con 0x55eeb33a8050 2024-03-24T00:42:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda4738c740 10 client.25934 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55eeb30213f0 con 0x55eeb33a8050 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.265+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda280d7c80 con 0x55eeb33a8050 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fda2c025020 con 0x55eeb33a8050 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 10 client.25934 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 20 client.25934 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb33bdea0 0x55eeb33c0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 4 client.25934 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eeb331a790 con 0x55eeb33bdea0 2024-03-24T00:42:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda41200640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb33bdea0 0x55eeb33c0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda27400640 10 client.25934 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.266+0000 7fda41200640 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb33bdea0 0x55eeb33c0320 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf6c8c9e8193e01 server_cookie=2396626d3dd6e24f in_seq=0 out_seq=0 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda27400640 1 -- 172.21.15.123:0/4030319450 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eeb331a790 con 0x55eeb33bdea0 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda27400640 10 client.25934 handle_command_reply: tid=0 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 client.25934 shutdown 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 msgr2=0x55eeb33b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fda2c012cf0 tx=0x7fda2c00f790 comp rx=0 tx=0).stop 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 shutdown_connections 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb33bdea0 0x55eeb33c0320 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8a40 0x55eeb33b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 --2- 172.21.15.123:0/4030319450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8050 0x55eeb33b2060 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.267+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 >> 172.21.15.123:0/4030319450 conn(0x55eeb3370200 msgr2=0x55eeb33a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.268+0000 7fda27400640 0 client.25934 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:00.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.268+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 shutdown_connections 2024-03-24T00:42:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.268+0000 7fda4738c740 1 -- 172.21.15.123:0/4030319450 wait complete. 2024-03-24T00:42:00.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.268+0000 7fda4738c740 20 client.25934 trim_cache size 0 max 16384 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.288+0000 7fda4738c740 1 Processor -- start 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.288+0000 7fda4738c740 1 -- start start 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a8aa0 con 0x55eeb33adbb0 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1ab060 con 0x55eeb33a8a40 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda4738c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a90a0 con 0x55eeb33a8050 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33698/0 (socket says 172.21.15.123:33698) 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47596/0 (socket says 172.21.15.123:47596) 2024-03-24T00:42:00.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41c00640 1 -- 172.21.15.123:0/3267002376 learned_addr learned my addr 172.21.15.123:0/3267002376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda42600640 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 -- 172.21.15.123:0/3267002376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 msgr2=0x55eeb339e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 -- 172.21.15.123:0/3267002376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 msgr2=0x55eeb33a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41200640 1 -- 172.21.15.123:0/3267002376 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeb2fe10f0 con 0x55eeb33a8a40 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.289+0000 7fda41c00640 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.290+0000 7fda41200640 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fda2809b630 tx=0x7fda28086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec427d88d1792915 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.290+0000 7fda32000640 1 -- 172.21.15.123:0/3267002376 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28097bd0 con 0x55eeb33a8a40 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.290+0000 7fda32000640 1 -- 172.21.15.123:0/3267002376 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda2809c030 con 0x55eeb33a8a40 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.290+0000 7fda32000640 1 -- 172.21.15.123:0/3267002376 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2809b290 con 0x55eeb33a8a40 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- 172.21.15.123:0/3267002376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 msgr2=0x55eeb33b95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fda2809b630 tx=0x7fda28086040 comp rx=0 tx=0).stop 2024-03-24T00:42:00.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- 172.21.15.123:0/3267002376 shutdown_connections 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b95a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 --2- 172.21.15.123:0/3267002376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb339e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- 172.21.15.123:0/3267002376 >> 172.21.15.123:0/3267002376 conn(0x55eeb3370200 msgr2=0x55eeb33a9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- 172.21.15.123:0/3267002376 shutdown_connections 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- 172.21.15.123:0/3267002376 wait complete. 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 -- start start 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.291+0000 7fda4738c740 1 Processor -- start 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb33b6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a96e0 con 0x55eeb33adbb0 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1aaa00 con 0x55eeb33a8a40 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a9d40 con 0x55eeb33a8050 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda4738c740 10 client.0 _ll_register_callbacks cb 0x7fda2c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda42600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47598/0 (socket says 172.21.15.123:47598) 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda41200640 1 -- 172.21.15.123:0/1894534180 learned_addr learned my addr 172.21.15.123:0/1894534180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.292+0000 7fda41c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb33b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 -- 172.21.15.123:0/1894534180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 msgr2=0x55eeb33b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb33b6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 -- 172.21.15.123:0/1894534180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 msgr2=0x55eeb33b7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 -- 172.21.15.123:0/1894534180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eeb33b8230 con 0x55eeb33a8a40 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda41200640 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c6a0 tx=0x7fda2809fb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503e83a3af7010ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda27400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2809b470 con 0x55eeb33a8a40 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fda280ccba0 con 0x55eeb33a8a40 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.293+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda2809c030 con 0x55eeb33a8a40 2024-03-24T00:42:00.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda2809b700 con 0x55eeb33a8a40 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 10 client.25940 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55eeb33b9540 con 0x55eeb33a8a40 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda2809c030 con 0x55eeb33a8a40 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fda2809bb90 con 0x55eeb33a8a40 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 10 client.25940 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 10 client.25940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 20 client.25940 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb34197b0 0x55eeb341bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 4 client.25940 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eeb3314ae0 con 0x55eeb34197b0 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.294+0000 7fda41c00640 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb34197b0 0x55eeb341bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda41c00640 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb34197b0 0x55eeb341bba0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f87fa1a924b3356 server_cookie=e7049b87a2ef0ac5 in_seq=0 out_seq=0 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda27400640 10 client.25940 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda27400640 1 -- 172.21.15.123:0/1894534180 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55eeb3314ae0 con 0x55eeb34197b0 2024-03-24T00:42:00.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda27400640 10 client.25940 handle_command_reply: tid=0 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 client.25940 shutdown 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 msgr2=0x55eeb33b6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c6a0 tx=0x7fda2809fb00 comp rx=0 tx=0).stop 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 shutdown_connections 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eeb33adbb0 0x55eeb33b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb33a8a40 0x55eeb33b6ca0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eeb34197b0 0x55eeb341bba0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 --2- 172.21.15.123:0/1894534180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eeb33a8050 0x55eeb33b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda27400640 0 client.25940 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.295+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 >> 172.21.15.123:0/1894534180 conn(0x55eeb3370200 msgr2=0x55eeb33a93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 shutdown_connections 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda4738c740 1 -- 172.21.15.123:0/1894534180 wait complete. 2024-03-24T00:42:00.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda4738c740 20 client.25940 trim_cache size 0 max 16384 2024-03-24T00:42:00.303 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:42:00.303 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:42:00.303 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:42:00.303 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:42:00.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 msgr2=0x7fda200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 0x7fda200a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda34005c40 tx=0x7fda3402c690 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 msgr2=0x7fda3c073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e9f0 tx=0x7fda2800edf0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 shutdown_connections 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda3c103150 0x7fda3c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda3c102790 0x7fda3c073a90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fda200a2800 0x7fda200a4cc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 --2- 172.21.15.123:0/344625471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3c1015a0 0x7fda3c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.296+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 >> 172.21.15.123:0/344625471 conn(0x7fda3c0fcb10 msgr2=0x7fda3c105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.297+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 shutdown_connections 2024-03-24T00:42:00.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.297+0000 7fda43000640 1 -- 172.21.15.123:0/344625471 wait complete. 2024-03-24T00:42:00.317 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:42:00.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.533+0000 7f5998000640 1 -- 172.21.15.123:0/576301671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5990102160 msgr2=0x7f5990104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.533+0000 7f5998000640 1 --2- 172.21.15.123:0/576301671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5990102160 0x7f5990104550 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5984009900 tx=0x7f598402bc30 comp rx=0 tx=0).stop 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 -- 172.21.15.123:0/576301671 shutdown_connections 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 --2- 172.21.15.123:0/576301671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59901073c0 0x7f5990109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 --2- 172.21.15.123:0/576301671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990104a90 0x7f5990106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 --2- 172.21.15.123:0/576301671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5990102160 0x7f5990104550 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 -- 172.21.15.123:0/576301671 >> 172.21.15.123:0/576301671 conn(0x7f59900fbb50 msgr2=0x7f59900fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 -- 172.21.15.123:0/576301671 shutdown_connections 2024-03-24T00:42:00.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.534+0000 7f5998000640 1 -- 172.21.15.123:0/576301671 wait complete. 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.535+0000 7f5998000640 1 Processor -- start 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.535+0000 7f5998000640 1 -- start start 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.535+0000 7f5998000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 0x7f5990199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.535+0000 7f5998000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5998000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59901073c0 0x7f59901a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5998000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599010d390 con 0x7f5990102160 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5998000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599010d210 con 0x7f59901073c0 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5998000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599010d510 con 0x7f5990104a90 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33722/0 (socket says 172.21.15.123:33722) 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 -- 172.21.15.123:0/89948888 learned_addr learned my addr 172.21.15.123:0/89948888 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5996000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59901073c0 0x7f59901a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 -- 172.21.15.123:0/89948888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59901073c0 msgr2=0x7f59901a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59901073c0 0x7f59901a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.536+0000 7f5994c00640 1 -- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 msgr2=0x7f5990199c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:42:00.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5995600640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 0x7f5990199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5994c00640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 0x7f5990199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5994c00640 1 -- 172.21.15.123:0/89948888 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59901a1870 con 0x7f5990104a90 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5995600640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 0x7f5990199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5994c00640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f597c00e9c0 tx=0x7f597c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c00cd80 con 0x7f5990104a90 2024-03-24T00:42:00.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59901a1b60 con 0x7f5990104a90 2024-03-24T00:42:00.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f597c005500 con 0x7f5990104a90 2024-03-24T00:42:00.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c010690 con 0x7f5990104a90 2024-03-24T00:42:00.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.537+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59901a2100 con 0x7f5990104a90 2024-03-24T00:42:00.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.539+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5990108f10 con 0x7f5990104a90 2024-03-24T00:42:00.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.542+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f597c0046c0 con 0x7f5990104a90 2024-03-24T00:42:00.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.544+0000 7f598e000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 0x7f59780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:00.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.544+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f597c0cef30 con 0x7f5990104a90 2024-03-24T00:42:00.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.544+0000 7f5995600640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 0x7f59780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:00.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.545+0000 7f5995600640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 0x7f59780a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5984036040 tx=0x7f5984000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:00.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.548+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f597c09bfe0 con 0x7f5990104a90 2024-03-24T00:42:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:00 smithi177 ceph-mon[36034]: pgmap v508: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 64 MiB/s wr, 6.57k op/s 2024-03-24T00:42:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:00 smithi204 ceph-mon[36345]: pgmap v508: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 64 MiB/s wr, 6.57k op/s 2024-03-24T00:42:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.833+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59900622a0 con 0x7f5990104a90 2024-03-24T00:42:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.834+0000 7f598e000640 1 -- 172.21.15.123:0/89948888 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f597c09b8e0 con 0x7f5990104a90 2024-03-24T00:42:00.835 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:42:00.835 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:42:00.836 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 msgr2=0x7f59780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 0x7f59780a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5984036040 tx=0x7f5984000bc0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 msgr2=0x7f599019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f597c00e9c0 tx=0x7f597c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 shutdown_connections 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59901073c0 0x7f59901a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5990104a90 0x7f599019a1a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f59780a2880 0x7f59780a4d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 --2- 172.21.15.123:0/89948888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5990102160 0x7f5990199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 >> 172.21.15.123:0/89948888 conn(0x7f59900fbb50 msgr2=0x7f59901018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 shutdown_connections 2024-03-24T00:42:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:00.837+0000 7f5998000640 1 -- 172.21.15.123:0/89948888 wait complete. 2024-03-24T00:42:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:00 smithi123 ceph-mon[31487]: pgmap v508: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 64 MiB/s wr, 6.57k op/s 2024-03-24T00:42:00.862 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:42:00.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 -- 172.21.15.123:0/975263683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 msgr2=0x7f6bf8106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 --2- 172.21.15.123:0/975263683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8106d80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009900 tx=0x7f6be402bc30 comp rx=0 tx=0).stop 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 -- 172.21.15.123:0/975263683 shutdown_connections 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 --2- 172.21.15.123:0/975263683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf81072c0 0x7f6bf81096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 --2- 172.21.15.123:0/975263683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8106d80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 --2- 172.21.15.123:0/975263683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8102060 0x7f6bf8104450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 -- 172.21.15.123:0/975263683 >> 172.21.15.123:0/975263683 conn(0x7f6bf80fba10 msgr2=0x7f6bf80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 -- 172.21.15.123:0/975263683 shutdown_connections 2024-03-24T00:42:01.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.078+0000 7f6bf7200640 1 -- 172.21.15.123:0/975263683 wait complete. 2024-03-24T00:42:01.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 Processor -- start 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 -- start start 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 0x7f6bf8197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 0x7f6bf819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf810d260 con 0x7f6bf8102060 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf810d0e0 con 0x7f6bf8104990 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.079+0000 7f6bf7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf810d3e0 con 0x7f6bf81072c0 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 0x7f6bf819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47638/0 (socket says 172.21.15.123:47638) 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf5400640 1 -- 172.21.15.123:0/4095507099 learned_addr learned my addr 172.21.15.123:0/4095507099 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 0x7f6bf8197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 -- 172.21.15.123:0/4095507099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 msgr2=0x7f6bf8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 -- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 msgr2=0x7f6bf8197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 0x7f6bf8197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf6800640 1 -- 172.21.15.123:0/4095507099 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf819f4d0 con 0x7f6bf81072c0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.080+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 0x7f6bf8197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6bf6800640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 0x7f6bf819ee60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000ec50 tx=0x7f6bf000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6beea00640 1 -- 172.21.15.123:0/4095507099 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0003b70 con 0x7f6bf81072c0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf819f7c0 con 0x7f6bf81072c0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6beea00640 1 -- 172.21.15.123:0/4095507099 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bf0003d10 con 0x7f6bf81072c0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6beea00640 1 -- 172.21.15.123:0/4095507099 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0010890 con 0x7f6bf81072c0 2024-03-24T00:42:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.081+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf819fce0 con 0x7f6bf81072c0 2024-03-24T00:42:01.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6beea00640 1 -- 172.21.15.123:0/4095507099 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6bf0003fc0 con 0x7f6bf81072c0 2024-03-24T00:42:01.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 Processor -- start 2024-03-24T00:42:01.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 -- start start 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc9168d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc916ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc90cd2c0 con 0x557cc916a4e0 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc9075c90 con 0x557cc9169b40 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.086+0000 7f6bff322740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc90c9360 con 0x557cc9168920 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47640/0 (socket says 172.21.15.123:47640) 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 -- 172.21.15.123:0/692143814 learned_addr learned my addr 172.21.15.123:0/692143814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5400640 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc916ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5400640 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc9168d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 -- 172.21.15.123:0/692143814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 msgr2=0x557cc9168d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc9168d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 -- 172.21.15.123:0/692143814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 msgr2=0x557cc916ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc916ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.087+0000 7f6bf5e00640 1 -- 172.21.15.123:0/692143814 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc909a080 con 0x557cc9169b40 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009d80 tx=0x7f6be80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b95ba7f87b7582a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6beea00640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 0x7f6bdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6be3000640 1 -- 172.21.15.123:0/692143814 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be801b670 con 0x557cc9169b40 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6be3000640 1 -- 172.21.15.123:0/692143814 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be8016070 con 0x557cc9169b40 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6be3000640 1 -- 172.21.15.123:0/692143814 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8012bc0 con 0x557cc9169b40 2024-03-24T00:42:01.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6beea00640 1 -- 172.21.15.123:0/4095507099 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6bf00cf020 con 0x7f6bf81072c0 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.088+0000 7f6bf5400640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 0x7f6bdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 -- 172.21.15.123:0/692143814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 msgr2=0x557cc9169fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009d80 tx=0x7f6be80117b0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bf5400640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 0x7f6bdc0a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6be4006010 tx=0x7f6be4005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 -- 172.21.15.123:0/692143814 shutdown_connections 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc916ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc9169fa0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 --2- 172.21.15.123:0/692143814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc9168d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 -- 172.21.15.123:0/692143814 >> 172.21.15.123:0/692143814 conn(0x557cc9163de0 msgr2=0x557cc91641e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 -- 172.21.15.123:0/692143814 shutdown_connections 2024-03-24T00:42:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.089+0000 7f6bff322740 1 -- 172.21.15.123:0/692143814 wait complete. 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.090+0000 7f6bff322740 1 -- start start 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.090+0000 7f6bff322740 1 Processor -- start 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc917b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc91829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc916b500 con 0x557cc916a4e0 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc90890d0 con 0x557cc9169b40 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc916b680 con 0x557cc9168920 2024-03-24T00:42:01.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bff322740 10 client.0 _ll_register_callbacks cb 0x557cc908de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc917b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc917b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47646/0 (socket says 172.21.15.123:47646) 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5400640 1 -- 172.21.15.123:0/2301154382 learned_addr learned my addr 172.21.15.123:0/2301154382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf6800640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc91829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 -- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 msgr2=0x557cc917b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc917b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 -- 172.21.15.123:0/2301154382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 msgr2=0x557cc91829f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc91829f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.091+0000 7f6bf5e00640 1 -- 172.21.15.123:0/2301154382 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cc8de13f0 con 0x557cc9168920 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917b470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6be8011cb0 tx=0x7f6be80045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13293c20a3b05833 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bd7400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8028020 con 0x557cc9168920 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6be8012d50 con 0x557cc9168920 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6be8016070 con 0x557cc9168920 2024-03-24T00:42:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be8009e70 con 0x557cc9168920 2024-03-24T00:42:01.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bff322740 10 client.25960 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:01.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.092+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557cc8eab7d0 con 0x557cc9168920 2024-03-24T00:42:01.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be8016070 con 0x557cc9168920 2024-03-24T00:42:01.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6be8023070 con 0x557cc9168920 2024-03-24T00:42:01.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 10 client.25960 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:01.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 10 client.25960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 20 client.25960 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc9186b50 0x557cc9188f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 4 client.25960 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cc90da800 con 0x557cc9186b50 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.093+0000 7f6bf5400640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc9186b50 0x557cc9188f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.094+0000 7f6bf5400640 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc9186b50 0x557cc9188f40 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfbc5942ce21e83b server_cookie=dccab4390cd4b671 in_seq=0 out_seq=0 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.094+0000 7f6bd7400640 10 client.25960 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bd7400640 1 -- 172.21.15.123:0/2301154382 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557cc90da800 con 0x557cc9186b50 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bd7400640 10 client.25960 handle_command_reply: tid=0 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 client.25960 shutdown 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 msgr2=0x557cc917b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917b470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6be8011cb0 tx=0x7f6be80045f0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 shutdown_connections 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc91829f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9169b40 0x557cc917b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc9186b50 0x557cc9188f40 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 --2- 172.21.15.123:0/2301154382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917b470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bd7400640 0 client.25960 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 >> 172.21.15.123:0/2301154382 conn(0x557cc9163de0 msgr2=0x557cc9167040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 shutdown_connections 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.095+0000 7f6bff322740 1 -- 172.21.15.123:0/2301154382 wait complete. 2024-03-24T00:42:01.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.096+0000 7f6bff322740 20 client.25960 trim_cache size 0 max 16384 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 Processor -- start 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 -- start start 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 0x557cc917a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc9162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a47b0 con 0x557cc916a4e0 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a6d70 con 0x557cc9168920 2024-03-24T00:42:01.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.112+0000 7f6bff322740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a4db0 con 0x557cc9169b40 2024-03-24T00:42:01.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc9162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33766/0 (socket says 172.21.15.123:33766) 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5400640 1 -- 172.21.15.123:0/4074942000 learned_addr learned my addr 172.21.15.123:0/4074942000 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 0x557cc917a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 -- 172.21.15.123:0/4074942000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 msgr2=0x557cc91829b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 -- 172.21.15.123:0/4074942000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 msgr2=0x557cc9162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc9162fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 -- 172.21.15.123:0/4074942000 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc9164d50 con 0x557cc9168920 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5400640 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.113+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 0x557cc917a360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8004490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16647d4ce52af7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.114+0000 7f6be2600640 1 -- 172.21.15.123:0/4074942000 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8028040 con 0x557cc9168920 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.114+0000 7f6be2600640 1 -- 172.21.15.123:0/4074942000 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be8016030 con 0x557cc9168920 2024-03-24T00:42:01.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.114+0000 7f6be2600640 1 -- 172.21.15.123:0/4074942000 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8032080 con 0x557cc9168920 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.114+0000 7f6bff322740 1 -- 172.21.15.123:0/4074942000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 msgr2=0x557cc917a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.114+0000 7f6bff322740 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 0x557cc917a360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8004490 comp rx=0 tx=0).stop 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 -- 172.21.15.123:0/4074942000 shutdown_connections 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc916a4e0 0x557cc9162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9169b40 0x557cc91829b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 --2- 172.21.15.123:0/4074942000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc9168920 0x557cc917a360 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 -- 172.21.15.123:0/4074942000 >> 172.21.15.123:0/4074942000 conn(0x557cc9163de0 msgr2=0x557cc91628d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 -- 172.21.15.123:0/4074942000 shutdown_connections 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 -- 172.21.15.123:0/4074942000 wait complete. 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 -- start start 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.115+0000 7f6bff322740 1 Processor -- start 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc9169b40 0x557cc917ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 0x557cc9181ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a53f0 con 0x557cc9169b40 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a6710 con 0x557cc916a4e0 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81a5a50 con 0x557cc9168920 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bff322740 10 client.0 _ll_register_callbacks cb 0x7f6be8009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33770/0 (socket says 172.21.15.123:33770) 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 -- 172.21.15.123:0/1747303949 learned_addr learned my addr 172.21.15.123:0/1747303949 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf6800640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 0x557cc9181ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5400640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc9169b40 0x557cc917ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 -- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 msgr2=0x557cc9181ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 0x557cc9181ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 -- 172.21.15.123:0/1747303949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc9169b40 msgr2=0x557cc917ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc9169b40 0x557cc917ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.116+0000 7f6bf5e00640 1 -- 172.21.15.123:0/1747303949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cc91822d0 con 0x557cc9168920 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bf6800640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 0x557cc9181ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bf5e00640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009c70 tx=0x7f6be8004420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fee11ed4dff829f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bd7400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8032070 con 0x557cc9168920 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6be80038f0 con 0x557cc9168920 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6be8016030 con 0x557cc9168920 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be8048050 con 0x557cc9168920 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bff322740 10 client.25966 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:01.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.117+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557cc91e29d0 con 0x557cc9168920 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be8016030 con 0x557cc9168920 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6be8023070 con 0x557cc9168920 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 10 client.25966 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 10 client.25966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 20 client.25966 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc91e3240 0x557cc91e5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 4 client.25966 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557cc90d4b50 con 0x557cc91e3240 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.118+0000 7f6bf5400640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc91e3240 0x557cc91e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.119+0000 7f6bf5400640 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc91e3240 0x557cc91e5660 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5338eb90309e8c server_cookie=d8134bab55501034 in_seq=0 out_seq=0 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.119+0000 7f6bd7400640 10 client.25966 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.119+0000 7f6bd7400640 1 -- 172.21.15.123:0/1747303949 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557cc90d4b50 con 0x557cc91e3240 2024-03-24T00:42:01.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.119+0000 7f6bd7400640 10 client.25966 handle_command_reply: tid=0 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.119+0000 7f6bff322740 1 client.25966 shutdown 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 msgr2=0x557cc917e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009c70 tx=0x7f6be8004420 comp rx=0 tx=0).stop 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 shutdown_connections 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557cc916a4e0 0x557cc9181ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557cc9169b40 0x557cc917ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557cc91e3240 0x557cc91e5660 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 --2- 172.21.15.123:0/1747303949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cc9168920 0x557cc917e780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bd7400640 0 client.25966 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 >> 172.21.15.123:0/1747303949 conn(0x557cc9163de0 msgr2=0x557cc918d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 shutdown_connections 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 1 -- 172.21.15.123:0/1747303949 wait complete. 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.120+0000 7f6bff322740 20 client.25966 trim_cache size 0 max 16384 2024-03-24T00:42:01.123 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:42:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 msgr2=0x7f6bdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 0x7f6bdc0a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6be4006010 tx=0x7f6be4005b50 comp rx=0 tx=0).stop 2024-03-24T00:42:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 msgr2=0x7f6bf819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 0x7f6bf819ee60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000ec50 tx=0x7f6bf000c6a0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 shutdown_connections 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf81072c0 0x7f6bf819ee60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bf8104990 0x7f6bf8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6bdc0a2880 0x7f6bdc0a4d40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 --2- 172.21.15.123:0/4095507099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf8102060 0x7f6bf8197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 >> 172.21.15.123:0/4095507099 conn(0x7f6bf80fba10 msgr2=0x7f6bf8102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 shutdown_connections 2024-03-24T00:42:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:01.121+0000 7f6bf7200640 1 -- 172.21.15.123:0/4095507099 wait complete. 2024-03-24T00:42:01.414 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:01 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/89948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:42:01.414 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:01 smithi123 ceph-mon[31487]: pgmap v509: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 62 MiB/s wr, 7.01k op/s 2024-03-24T00:42:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:01 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/89948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:42:01.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:01 smithi177 ceph-mon[36034]: pgmap v509: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 62 MiB/s wr, 7.01k op/s 2024-03-24T00:42:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:01 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/89948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:42:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:01 smithi204 ceph-mon[36345]: pgmap v509: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 62 MiB/s wr, 7.01k op/s 2024-03-24T00:42:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 -- 172.21.15.123:0/3122559614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 msgr2=0x7f6724104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 --2- 172.21.15.123:0/3122559614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f6724104f40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6710009900 tx=0x7f671002bc30 comp rx=0 tx=0).stop 2024-03-24T00:42:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 -- 172.21.15.123:0/3122559614 shutdown_connections 2024-03-24T00:42:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 --2- 172.21.15.123:0/3122559614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 0x7f672410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 --2- 172.21.15.123:0/3122559614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f6724104f40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 --2- 172.21.15.123:0/3122559614 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 -- 172.21.15.123:0/3122559614 >> 172.21.15.123:0/3122559614 conn(0x7f67240fee60 msgr2=0x7f6724101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 -- 172.21.15.123:0/3122559614 shutdown_connections 2024-03-24T00:42:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.932+0000 7f672bc00640 1 -- 172.21.15.123:0/3122559614 wait complete. 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.933+0000 7f672bc00640 1 Processor -- start 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.933+0000 7f672bc00640 1 -- start start 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 0x7f67241a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672410f530 con 0x7f6724105480 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672410f3b0 con 0x7f6724104ac0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f672bc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672410f6b0 con 0x7f67241038d0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6729200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47688/0 (socket says 172.21.15.123:47688) 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 -- 172.21.15.123:0/2888687358 learned_addr learned my addr 172.21.15.123:0/2888687358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6729c00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 0x7f67241a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 -- 172.21.15.123:0/2888687358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 msgr2=0x7f6724199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 -- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 msgr2=0x7f67241a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 0x7f67241a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6723e00640 1 -- 172.21.15.123:0/2888687358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67241a1740 con 0x7f6724104ac0 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.934+0000 7f6729200640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f6723e00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6710009230 tx=0x7f671002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f6721600640 1 -- 172.21.15.123:0/2888687358 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710041070 con 0x7f6724104ac0 2024-03-24T00:42:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f6721600640 1 -- 172.21.15.123:0/2888687358 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6710004450 con 0x7f6724104ac0 2024-03-24T00:42:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f6721600640 1 -- 172.21.15.123:0/2888687358 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710003e20 con 0x7f6724104ac0 2024-03-24T00:42:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67241a19d0 con 0x7f6724104ac0 2024-03-24T00:42:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.935+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67241a1ec0 con 0x7f6724104ac0 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.938+0000 7f6721600640 1 -- 172.21.15.123:0/2888687358 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6710044070 con 0x7f6724104ac0 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.938+0000 7f672e9b6740 1 Processor -- start 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 -- start start 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1db66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1db91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1d743b0 con 0x5626a1db6d40 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1d16830 con 0x5626a1db1bd0 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f672e9b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1cbf5d0 con 0x5626a1db11e0 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33810/0 (socket says 172.21.15.123:33810) 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 -- 172.21.15.123:0/2947498537 learned_addr learned my addr 172.21.15.123:0/2947498537 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729200640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1db66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6723e00640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1db91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 -- 172.21.15.123:0/2947498537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 msgr2=0x5626a1db66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1db66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 -- 172.21.15.123:0/2947498537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 msgr2=0x5626a1db91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.939+0000 7f6729c00640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1db91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6729c00640 1 -- 172.21.15.123:0/2947498537 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626a1cd72c0 con 0x5626a1db11e0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6723e00640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1db91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6729c00640 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f671c009fd0 tx=0x7f671c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553f67e99681e409 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6716000640 1 -- 172.21.15.123:0/2947498537 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c004d20 con 0x5626a1db11e0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6716000640 1 -- 172.21.15.123:0/2947498537 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f671c004050 con 0x5626a1db11e0 2024-03-24T00:42:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6716000640 1 -- 172.21.15.123:0/2947498537 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c005220 con 0x5626a1db11e0 2024-03-24T00:42:01.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.940+0000 7f6721600640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 0x7f670c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f6721600640 1 -- 172.21.15.123:0/2888687358 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67100f0650 con 0x7f6724104ac0 2024-03-24T00:42:01.945 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f6729200640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 0x7f670c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 -- 172.21.15.123:0/2947498537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 msgr2=0x5626a1db1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f671c009fd0 tx=0x7f671c0118d0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f6729200640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 0x7f670c0a4d60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5626a1db1fd0 tx=0x7f6718005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 -- 172.21.15.123:0/2947498537 shutdown_connections 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1db91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1db66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2947498537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db11e0 0x5626a1db1600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 -- 172.21.15.123:0/2947498537 >> 172.21.15.123:0/2947498537 conn(0x5626a1dac750 msgr2=0x5626a1dacb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.941+0000 7f672e9b6740 1 -- 172.21.15.123:0/2947498537 shutdown_connections 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.942+0000 7f672e9b6740 1 -- 172.21.15.123:0/2947498537 wait complete. 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.942+0000 7f672e9b6740 1 -- start start 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.942+0000 7f672e9b6740 1 Processor -- start 2024-03-24T00:42:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dbb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1dc2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1cd2a20 con 0x5626a1db6d40 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1d128d0 con 0x5626a1db11e0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626a1db9cf0 con 0x5626a1db1bd0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f6729200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f6729200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47714/0 (socket says 172.21.15.123:47714) 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f6729200640 1 -- 172.21.15.123:0/827198325 learned_addr learned my addr 172.21.15.123:0/827198325 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f672e9b6740 10 client.0 _ll_register_callbacks cb 0x5626a1d0f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f6723e00640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dbb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.943+0000 7f6729c00640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1dc2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 -- 172.21.15.123:0/827198325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 msgr2=0x5626a1dbb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dbb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 -- 172.21.15.123:0/827198325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 msgr2=0x5626a1dc2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1dc2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 -- 172.21.15.123:0/827198325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626a1a2b210 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6723e00640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dbb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f6729200640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6718004620 tx=0x7f6718004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46075fb9841ee68d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f670aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671800f8f0 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f671800fa90 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.944+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f671c004050 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f671800fc90 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 10 client.25976 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5626a1af7620 con 0x5626a1db11e0 2024-03-24T00:42:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f671c004050 con 0x5626a1db11e0 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6718027070 con 0x5626a1db11e0 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 10 client.25976 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 10 client.25976 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 20 client.25976 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1dc7190 0x5626a1dc9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 4 client.25976 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.945+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626a1d23d70 con 0x5626a1dc7190 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.946+0000 7f6723e00640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1dc7190 0x5626a1dc9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.946+0000 7f6723e00640 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1dc7190 0x5626a1dc9580 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72da448ee42ebc64 server_cookie=2be08d5a173e9a52 in_seq=0 out_seq=0 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.946+0000 7f670aa00640 10 client.25976 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f670aa00640 1 -- 172.21.15.123:0/827198325 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626a1d23d70 con 0x5626a1dc7190 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f670aa00640 10 client.25976 handle_command_reply: tid=0 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 client.25976 shutdown 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 msgr2=0x5626a1dbb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6718004620 tx=0x7f6718004730 comp rx=0 tx=0).stop 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 shutdown_connections 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db6d40 0x5626a1dc2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1dc7190 0x5626a1dc9580 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dbb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 --2- 172.21.15.123:0/827198325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db11e0 0x5626a1dbb3c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 >> 172.21.15.123:0/827198325 conn(0x5626a1dac750 msgr2=0x5626a1dab950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f670aa00640 0 client.25976 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 shutdown_connections 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 1 -- 172.21.15.123:0/827198325 wait complete. 2024-03-24T00:42:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.947+0000 7f672e9b6740 20 client.25976 trim_cache size 0 max 16384 2024-03-24T00:42:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 Processor -- start 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 -- start start 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dbafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db6d40 0x5626a1dbb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241a8c60 con 0x5626a1db11e0 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241ab220 con 0x5626a1db6d40 2024-03-24T00:42:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f672e9b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241a9260 con 0x5626a1db1bd0 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f6723e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f6729c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db6d40 0x5626a1dbb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f6723e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33826/0 (socket says 172.21.15.123:33826) 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f6723e00640 1 -- 172.21.15.123:0/2320074572 learned_addr learned my addr 172.21.15.123:0/2320074572 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.965+0000 7f6729200640 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dbafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 -- 172.21.15.123:0/2320074572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db6d40 msgr2=0x5626a1dbb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db6d40 0x5626a1dbb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 -- 172.21.15.123:0/2320074572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 msgr2=0x5626a1dbafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dbafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 -- 172.21.15.123:0/2320074572 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626a1dc98f0 con 0x5626a1db1bd0 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6729200640 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dbafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6723e00640 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f67100bcfe0 tx=0x7f67100edfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7785e312101168c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6715600640 1 -- 172.21.15.123:0/2320074572 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671004b040 con 0x5626a1db1bd0 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6715600640 1 -- 172.21.15.123:0/2320074572 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67100c10f0 con 0x5626a1db1bd0 2024-03-24T00:42:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.966+0000 7f6715600640 1 -- 172.21.15.123:0/2320074572 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67100b96c0 con 0x5626a1db1bd0 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- 172.21.15.123:0/2320074572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 msgr2=0x5626a1dc2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f67100bcfe0 tx=0x7f67100edfd0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- 172.21.15.123:0/2320074572 shutdown_connections 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db6d40 0x5626a1dbb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db1bd0 0x5626a1dc2900 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 --2- 172.21.15.123:0/2320074572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dbafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- 172.21.15.123:0/2320074572 >> 172.21.15.123:0/2320074572 conn(0x5626a1dac750 msgr2=0x5626a1db2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- 172.21.15.123:0/2320074572 shutdown_connections 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- 172.21.15.123:0/2320074572 wait complete. 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.967+0000 7f672e9b6740 1 -- start start 2024-03-24T00:42:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 Processor -- start 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dc0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 0x5626a1dc1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241a98a0 con 0x5626a1db11e0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241aabc0 con 0x5626a1db1bd0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67241a9f00 con 0x5626a1db6d40 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f672e9b6740 10 client.0 _ll_register_callbacks cb 0x7f6718009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f6723e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f6729200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dc0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f6723e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47732/0 (socket says 172.21.15.123:47732) 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f6723e00640 1 -- 172.21.15.123:0/3404384855 learned_addr learned my addr 172.21.15.123:0/3404384855 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.968+0000 7f6729c00640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 0x5626a1dc1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 -- 172.21.15.123:0/3404384855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 msgr2=0x5626a1dc1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 0x5626a1dc1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 -- 172.21.15.123:0/3404384855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 msgr2=0x5626a1dc0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dc0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 -- 172.21.15.123:0/3404384855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626a1dc1c70 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6729c00640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 0x5626a1dc1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f6723e00640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67100b9a20 tx=0x7f6710009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4b38086d6aba6b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f670aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671004b070 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67100b96c0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.969+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67100c10f0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67100bd2b0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 10 client.25982 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5626a1e22ab0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67100c10f0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f67100b7bc0 con 0x5626a1db1bd0 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 10 client.25982 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 10 client.25982 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 20 client.25982 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1e23310 0x5626a1e25750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 4 client.25982 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5626a1d1e0c0 con 0x5626a1e23310 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.970+0000 7f6729200640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1e23310 0x5626a1e25750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.971+0000 7f6729200640 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1e23310 0x5626a1e25750 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d02d27334840a6d server_cookie=5d85566ec3c72f9d in_seq=0 out_seq=0 2024-03-24T00:42:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:01.971+0000 7f670aa00640 10 client.25982 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:02.140 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:42:02.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:02.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:42:02.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.417+0000 7f769ac00640 1 -- 172.21.15.123:0/4227838390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 msgr2=0x7f7694102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:02.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.417+0000 7f769ac00640 1 --2- 172.21.15.123:0/4227838390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694102c10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7688009860 tx=0x7f768802bb90 comp rx=0 tx=0).stop 2024-03-24T00:42:02.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.418+0000 7f769ac00640 1 -- 172.21.15.123:0/4227838390 shutdown_connections 2024-03-24T00:42:02.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.418+0000 7f769ac00640 1 --2- 172.21.15.123:0/4227838390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 0x7f76941099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.418+0000 7f769ac00640 1 --2- 172.21.15.123:0/4227838390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694102c10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.418+0000 7f769ac00640 1 --2- 172.21.15.123:0/4227838390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 0x7f76941019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.418+0000 7f769ac00640 1 -- 172.21.15.123:0/4227838390 >> 172.21.15.123:0/4227838390 conn(0x7f76940fcb10 msgr2=0x7f76940fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:02.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.419+0000 7f769ac00640 1 -- 172.21.15.123:0/4227838390 shutdown_connections 2024-03-24T00:42:02.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.420+0000 7f769ac00640 1 -- 172.21.15.123:0/4227838390 wait complete. 2024-03-24T00:42:02.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.423+0000 7f769ac00640 1 Processor -- start 2024-03-24T00:42:02.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 -- start start 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 0x7f7694110a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 0x7f7694111e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410b040 con 0x7f7694103150 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410aec0 con 0x7f76941015a0 2024-03-24T00:42:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.425+0000 7f769ac00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410b1c0 con 0x7f7694102790 2024-03-24T00:42:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.426+0000 7f7698e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.426+0000 7f7698e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33860/0 (socket says 172.21.15.123:33860) 2024-03-24T00:42:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.426+0000 7f7698e00640 1 -- 172.21.15.123:0/1139026863 learned_addr learned my addr 172.21.15.123:0/1139026863 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.426+0000 7f769a200640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 0x7f7694111e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.427+0000 7f7699800640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 0x7f7694110a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.428+0000 7f7698e00640 1 -- 172.21.15.123:0/1139026863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 msgr2=0x7f7694110a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:02.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.428+0000 7f7698e00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 0x7f7694110a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.428+0000 7f7698e00640 1 -- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 msgr2=0x7f7694111e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:02.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.428+0000 7f7698e00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 0x7f7694111e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.428+0000 7f7698e00640 1 -- 172.21.15.123:0/1139026863 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7694112580 con 0x7f7694102790 2024-03-24T00:42:02.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.429+0000 7f7698e00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7688009780 tx=0x7f7688004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:02.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.430+0000 7f7692000640 1 -- 172.21.15.123:0/1139026863 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688041020 con 0x7f7694102790 2024-03-24T00:42:02.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.430+0000 7f7692000640 1 -- 172.21.15.123:0/1139026863 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76880347b0 con 0x7f7694102790 2024-03-24T00:42:02.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.430+0000 7f7692000640 1 -- 172.21.15.123:0/1139026863 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688039040 con 0x7f7694102790 2024-03-24T00:42:02.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.430+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7694112810 con 0x7f7694102790 2024-03-24T00:42:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.433+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941aebe0 con 0x7f7694102790 2024-03-24T00:42:02.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.444+0000 7f7692000640 1 -- 172.21.15.123:0/1139026863 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7688044020 con 0x7f7694102790 2024-03-24T00:42:02.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.447+0000 7f7692000640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 0x7f767c0a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.447+0000 7f7692000640 1 -- 172.21.15.123:0/1139026863 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f76880efd30 con 0x7f7694102790 2024-03-24T00:42:02.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.451+0000 7f7699800640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 0x7f767c0a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.452+0000 7f7699800640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 0x7f767c0a4d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7694110760 tx=0x7f7680009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:02.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.454+0000 7f769ef89740 1 Processor -- start 2024-03-24T00:42:02.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.454+0000 7f769ef89740 1 -- start start 2024-03-24T00:42:02.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.456+0000 7f769ef89740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f769ef89740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c70a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f769ef89740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c70d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f769ef89740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c66aeb0 con 0x55f38c70ae10 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f769ef89740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c613a60 con 0x55f38c705ca0 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f769ef89740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c666f50 con 0x55f38c7052b0 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f7699800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f7699800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33868/0 (socket says 172.21.15.123:33868) 2024-03-24T00:42:02.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.457+0000 7f7699800640 1 -- 172.21.15.123:0/1400362151 learned_addr learned my addr 172.21.15.123:0/1400362151 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:02.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.459+0000 7f7699800640 1 -- 172.21.15.123:0/1400362151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 msgr2=0x55f38c70a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:42:02.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.459+0000 7f7699800640 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c70a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.459+0000 7f7699800640 1 -- 172.21.15.123:0/1400362151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 msgr2=0x55f38c70d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:02.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.459+0000 7f7699800640 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c70d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.459+0000 7f7699800640 1 -- 172.21.15.123:0/1400362151 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f38c4477e0 con 0x55f38c7052b0 2024-03-24T00:42:02.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.460+0000 7f7699800640 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7680011f50 tx=0x7f7680014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ef780ba3a8438d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:02.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.460+0000 7f7686a00640 1 -- 172.21.15.123:0/1400362151 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768001f620 con 0x55f38c7052b0 2024-03-24T00:42:02.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.461+0000 7f7686a00640 1 -- 172.21.15.123:0/1400362151 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f768001fca0 con 0x55f38c7052b0 2024-03-24T00:42:02.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.463+0000 7f7686a00640 1 -- 172.21.15.123:0/1400362151 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768001f620 con 0x55f38c7052b0 2024-03-24T00:42:02.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.463+0000 7f769ef89740 1 -- 172.21.15.123:0/1400362151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 msgr2=0x55f38c7056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:02.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.463+0000 7f769ef89740 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7680011f50 tx=0x7f7680014000 comp rx=0 tx=0).stop 2024-03-24T00:42:02.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.464+0000 7f769ef89740 1 -- 172.21.15.123:0/1400362151 shutdown_connections 2024-03-24T00:42:02.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.464+0000 7f769ef89740 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c70d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.464+0000 7f769ef89740 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c70a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.464+0000 7f769ef89740 1 --2- 172.21.15.123:0/1400362151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c7056d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.464+0000 7f769ef89740 1 -- 172.21.15.123:0/1400362151 >> 172.21.15.123:0/1400362151 conn(0x55f38c700820 msgr2=0x55f38c700c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:02.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.465+0000 7f769ef89740 1 -- 172.21.15.123:0/1400362151 shutdown_connections 2024-03-24T00:42:02.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.465+0000 7f769ef89740 1 -- 172.21.15.123:0/1400362151 wait complete. 2024-03-24T00:42:02.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.466+0000 7f769ef89740 1 -- start start 2024-03-24T00:42:02.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.466+0000 7f769ef89740 1 Processor -- start 2024-03-24T00:42:02.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.469+0000 7f769ef89740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c7052b0 0x55f38c70f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.469+0000 7f769ef89740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.469+0000 7f769ef89740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c716860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f7698e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f769ef89740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c70de10 con 0x55f38c70ae10 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f769ef89740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c626eb0 con 0x55f38c7052b0 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f769ef89740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38c70df90 con 0x55f38c705ca0 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f7698e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33874/0 (socket says 172.21.15.123:33874) 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f7698e00640 1 -- 172.21.15.123:0/3784838664 learned_addr learned my addr 172.21.15.123:0/3784838664 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:02.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.470+0000 7f769ef89740 10 client.0 _ll_register_callbacks cb 0x55f38c61b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.471+0000 7f7699800640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c7052b0 0x55f38c70f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f769a200640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c716860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 -- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c7052b0 msgr2=0x55f38c70f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c7052b0 0x55f38c70f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 -- 172.21.15.123:0/3784838664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 msgr2=0x55f38c716860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c716860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 -- 172.21.15.123:0/3784838664 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f38c4a7f70 con 0x55f38c705ca0 2024-03-24T00:42:02.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f769a200640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c716860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f7698e00640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f76880b9020 tx=0x7f768802bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d514c4f138f2a65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.472+0000 7f767b400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.473+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880a7070 con 0x55f38c705ca0 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.473+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f76880a5be0 con 0x55f38c705ca0 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.473+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f768001fca0 con 0x55f38c705ca0 2024-03-24T00:42:02.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.473+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76880a5e60 con 0x55f38c705ca0 2024-03-24T00:42:02.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.474+0000 7f769ef89740 10 client.25996 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:02.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.474+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f38c33f170 con 0x55f38c705ca0 2024-03-24T00:42:02.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.474+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f768001fca0 con 0x55f38c705ca0 2024-03-24T00:42:02.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.474+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f768803c030 con 0x55f38c705ca0 2024-03-24T00:42:02.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.474+0000 7f769ef89740 10 client.25996 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:02.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.477+0000 7f769ef89740 10 client.25996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:02.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.477+0000 7f769ef89740 20 client.25996 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:02.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.477+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c71b0e0 0x55f38c71d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:02.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.477+0000 7f769ef89740 4 client.25996 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:02.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.477+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f38c6783f0 con 0x55f38c71b0e0 2024-03-24T00:42:02.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.478+0000 7f7699800640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c71b0e0 0x55f38c71d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:02.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.480+0000 7f767b400640 10 client.25996 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:02.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:02.480+0000 7f7699800640 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c71b0e0 0x55f38c71d4d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de69bdd966baa200 server_cookie=d6ff20283337723b in_seq=0 out_seq=0 2024-03-24T00:42:04.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:04 smithi204 ceph-mon[36345]: pgmap v510: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 58 MiB/s wr, 6.95k op/s 2024-03-24T00:42:04.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:04 smithi123 ceph-mon[31487]: pgmap v510: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 58 MiB/s wr, 6.95k op/s 2024-03-24T00:42:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:04 smithi177 ceph-mon[36034]: pgmap v510: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 58 MiB/s wr, 6.95k op/s 2024-03-24T00:42:06.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:06 smithi204 ceph-mon[36345]: pgmap v511: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 55 MiB/s wr, 6.69k op/s 2024-03-24T00:42:06.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:06 smithi123 ceph-mon[31487]: pgmap v511: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 55 MiB/s wr, 6.69k op/s 2024-03-24T00:42:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:06 smithi177 ceph-mon[36034]: pgmap v511: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 55 MiB/s wr, 6.69k op/s 2024-03-24T00:42:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:08 smithi204 ceph-mon[36345]: pgmap v512: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 46 MiB/s wr, 5.91k op/s 2024-03-24T00:42:08.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:08 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:08.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:08 smithi123 ceph-mon[31487]: pgmap v512: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 46 MiB/s wr, 5.91k op/s 2024-03-24T00:42:08.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:08 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:08 smithi177 ceph-mon[36034]: pgmap v512: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 46 MiB/s wr, 5.91k op/s 2024-03-24T00:42:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:08 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:09.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:42:09.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:09.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:09.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:42:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:09.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:42:10.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:10 smithi204 ceph-mon[36345]: pgmap v513: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 50 MiB/s wr, 6.42k op/s 2024-03-24T00:42:10.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:10 smithi123 ceph-mon[31487]: pgmap v513: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 50 MiB/s wr, 6.42k op/s 2024-03-24T00:42:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:10 smithi177 ceph-mon[36034]: pgmap v513: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 50 MiB/s wr, 6.42k op/s 2024-03-24T00:42:12.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:12 smithi123 ceph-mon[31487]: pgmap v514: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 54 MiB/s wr, 6.78k op/s 2024-03-24T00:42:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:12 smithi177 ceph-mon[36034]: pgmap v514: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 54 MiB/s wr, 6.78k op/s 2024-03-24T00:42:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:12 smithi204 ceph-mon[36345]: pgmap v514: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 54 MiB/s wr, 6.78k op/s 2024-03-24T00:42:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:14 smithi123 ceph-mon[31487]: pgmap v515: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 50 MiB/s wr, 5.75k op/s 2024-03-24T00:42:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:14 smithi177 ceph-mon[36034]: pgmap v515: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 50 MiB/s wr, 5.75k op/s 2024-03-24T00:42:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:14 smithi204 ceph-mon[36345]: pgmap v515: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 50 MiB/s wr, 5.75k op/s 2024-03-24T00:42:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:16 smithi123 ceph-mon[31487]: pgmap v516: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 51 MiB/s wr, 5.24k op/s 2024-03-24T00:42:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:16 smithi177 ceph-mon[36034]: pgmap v516: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 51 MiB/s wr, 5.24k op/s 2024-03-24T00:42:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:16 smithi204 ceph-mon[36345]: pgmap v516: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 51 MiB/s wr, 5.24k op/s 2024-03-24T00:42:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:18 smithi123 ceph-mon[31487]: pgmap v517: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 47 MiB/s wr, 4.62k op/s 2024-03-24T00:42:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:18 smithi177 ceph-mon[36034]: pgmap v517: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 47 MiB/s wr, 4.62k op/s 2024-03-24T00:42:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:18 smithi204 ceph-mon[36345]: pgmap v517: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 47 MiB/s wr, 4.62k op/s 2024-03-24T00:42:20.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:20 smithi123 ceph-mon[31487]: pgmap v518: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 56 MiB/s wr, 5.19k op/s 2024-03-24T00:42:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:20 smithi177 ceph-mon[36034]: pgmap v518: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 56 MiB/s wr, 5.19k op/s 2024-03-24T00:42:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:20 smithi204 ceph-mon[36345]: pgmap v518: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 56 MiB/s wr, 5.19k op/s 2024-03-24T00:42:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:22 smithi177 ceph-mon[36034]: pgmap v519: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 67 MiB/s wr, 6.04k op/s 2024-03-24T00:42:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:22 smithi204 ceph-mon[36345]: pgmap v519: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 67 MiB/s wr, 6.04k op/s 2024-03-24T00:42:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:22 smithi123 ceph-mon[31487]: pgmap v519: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 67 MiB/s wr, 6.04k op/s 2024-03-24T00:42:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:24 smithi177 ceph-mon[36034]: pgmap v520: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 64 MiB/s wr, 5.88k op/s 2024-03-24T00:42:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:24 smithi204 ceph-mon[36345]: pgmap v520: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 64 MiB/s wr, 5.88k op/s 2024-03-24T00:42:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:24 smithi123 ceph-mon[31487]: pgmap v520: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 64 MiB/s wr, 5.88k op/s 2024-03-24T00:42:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:26 smithi177 ceph-mon[36034]: pgmap v521: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 64 MiB/s wr, 6.04k op/s 2024-03-24T00:42:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:26 smithi204 ceph-mon[36345]: pgmap v521: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 64 MiB/s wr, 6.04k op/s 2024-03-24T00:42:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:26 smithi123 ceph-mon[31487]: pgmap v521: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 64 MiB/s wr, 6.04k op/s 2024-03-24T00:42:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:28 smithi177 ceph-mon[36034]: pgmap v522: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 59 MiB/s wr, 5.61k op/s 2024-03-24T00:42:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:28 smithi204 ceph-mon[36345]: pgmap v522: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 59 MiB/s wr, 5.61k op/s 2024-03-24T00:42:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:28 smithi123 ceph-mon[31487]: pgmap v522: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 59 MiB/s wr, 5.61k op/s 2024-03-24T00:42:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:30 smithi177 ceph-mon[36034]: pgmap v523: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 66 MiB/s wr, 6.18k op/s 2024-03-24T00:42:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:30 smithi204 ceph-mon[36345]: pgmap v523: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 66 MiB/s wr, 6.18k op/s 2024-03-24T00:42:30.862 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:30 smithi123 ceph-mon[31487]: pgmap v523: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 66 MiB/s wr, 6.18k op/s 2024-03-24T00:42:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:32 smithi177 ceph-mon[36034]: pgmap v524: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 72 MiB/s wr, 6.62k op/s 2024-03-24T00:42:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:32 smithi204 ceph-mon[36345]: pgmap v524: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 72 MiB/s wr, 6.62k op/s 2024-03-24T00:42:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:32 smithi123 ceph-mon[31487]: pgmap v524: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 72 MiB/s wr, 6.62k op/s 2024-03-24T00:42:33.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:33 smithi177 ceph-mon[36034]: pgmap v525: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 67 MiB/s wr, 6.26k op/s 2024-03-24T00:42:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:33 smithi204 ceph-mon[36345]: pgmap v525: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 67 MiB/s wr, 6.26k op/s 2024-03-24T00:42:33.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:33 smithi123 ceph-mon[31487]: pgmap v525: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 67 MiB/s wr, 6.26k op/s 2024-03-24T00:42:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.436+0000 7f670aa00640 1 -- 172.21.15.123:0/3404384855 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5626a1d1e0c0 con 0x5626a1e23310 2024-03-24T00:42:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.436+0000 7f670aa00640 10 client.25982 handle_command_reply: tid=0 2024-03-24T00:42:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.436+0000 7f672e9b6740 1 client.25982 shutdown 2024-03-24T00:42:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 msgr2=0x5626a1dc06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67100b9a20 tx=0x7f6710009680 comp rx=0 tx=0).stop 2024-03-24T00:42:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 shutdown_connections 2024-03-24T00:42:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5626a1e23310 0x5626a1e25750 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5626a1db6d40 0x5626a1dc1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5626a1db1bd0 0x5626a1dc06c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 --2- 172.21.15.123:0/3404384855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5626a1db11e0 0x5626a1dc0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 >> 172.21.15.123:0/3404384855 conn(0x5626a1dac750 msgr2=0x5626a1dbafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f670aa00640 0 client.25982 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f767b400640 1 -- 172.21.15.123:0/3784838664 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f38c6783f0 con 0x55f38c71b0e0 2024-03-24T00:42:34.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.437+0000 7f767b400640 10 client.25996 handle_command_reply: tid=0 2024-03-24T00:42:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 shutdown_connections 2024-03-24T00:42:34.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f769ef89740 1 client.25996 shutdown 2024-03-24T00:42:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f672e9b6740 1 -- 172.21.15.123:0/3404384855 wait complete. 2024-03-24T00:42:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f672e9b6740 20 client.25982 trim_cache size 0 max 16384 2024-03-24T00:42:34.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 msgr2=0x55f38c70f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.438+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f76880b9020 tx=0x7f768802bb90 comp rx=0 tx=0).stop 2024-03-24T00:42:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:42:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:42:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:42:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 shutdown_connections 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c70ae10 0x55f38c716860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c71b0e0 0x55f38c71d4d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c705ca0 0x55f38c70f820 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 --2- 172.21.15.123:0/3784838664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c7052b0 0x55f38c70f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 >> 172.21.15.123:0/3784838664 conn(0x55f38c700820 msgr2=0x55f38c6ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f767b400640 0 client.25996 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 shutdown_connections 2024-03-24T00:42:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 msgr2=0x7f670c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 0x7f670c0a4d60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5626a1db1fd0 tx=0x7f6718005f90 comp rx=0 tx=0).stop 2024-03-24T00:42:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 msgr2=0x7f672419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6710009230 tx=0x7f671002c650 comp rx=0 tx=0).stop 2024-03-24T00:42:34.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 1 -- 172.21.15.123:0/3784838664 wait complete. 2024-03-24T00:42:34.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.439+0000 7f769ef89740 20 client.25996 trim_cache size 0 max 16384 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 shutdown_connections 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f670c0a28a0 0x7f670c0a4d60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724105480 0x7f67241a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6724104ac0 0x7f672419a060 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 --2- 172.21.15.123:0/2888687358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67241038d0 0x7f6724199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 >> 172.21.15.123:0/2888687358 conn(0x7f67240fee60 msgr2=0x7f6724107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 shutdown_connections 2024-03-24T00:42:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:34.440+0000 7f672bc00640 1 -- 172.21.15.123:0/2888687358 wait complete. 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 Processor -- start 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 -- start start 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c706e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c705ca0 0x55f38c716820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b57b0 con 0x55f38c705ca0 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b7d70 con 0x55f38c70ae10 2024-03-24T00:42:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.464+0000 7f769ef89740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b5db0 con 0x55f38c7052b0 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f7699800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c706e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59762/0 (socket says 172.21.15.123:59762) 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f7698e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c705ca0 0x55f38c716820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 -- 172.21.15.123:0/3938182818 learned_addr learned my addr 172.21.15.123:0/3938182818 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 -- 172.21.15.123:0/3938182818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 msgr2=0x55f38c706e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c706e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 -- 172.21.15.123:0/3938182818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c705ca0 msgr2=0x55f38c716820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c705ca0 0x55f38c716820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f769a200640 1 -- 172.21.15.123:0/3938182818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f38c70e3c0 con 0x55f38c70ae10 2024-03-24T00:42:34.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.465+0000 7f7699800640 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c706e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.466+0000 7f769a200640 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f768c009d50 tx=0x7f768c011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df932ef4f19667a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.466+0000 7f7686000640 1 -- 172.21.15.123:0/3938182818 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c01a3f0 con 0x55f38c70ae10 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.466+0000 7f7686000640 1 -- 172.21.15.123:0/3938182818 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f768c01aa70 con 0x55f38c70ae10 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.466+0000 7f7686000640 1 -- 172.21.15.123:0/3938182818 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c012be0 con 0x55f38c70ae10 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 -- 172.21.15.123:0/3938182818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 msgr2=0x55f38c721d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f768c009d50 tx=0x7f768c011750 comp rx=0 tx=0).stop 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 -- 172.21.15.123:0/3938182818 shutdown_connections 2024-03-24T00:42:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c70ae10 0x55f38c721d50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c705ca0 0x55f38c716820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 --2- 172.21.15.123:0/3938182818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c7052b0 0x55f38c706e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 -- 172.21.15.123:0/3938182818 >> 172.21.15.123:0/3938182818 conn(0x55f38c700820 msgr2=0x55f38c706710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 -- 172.21.15.123:0/3938182818 shutdown_connections 2024-03-24T00:42:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.467+0000 7f769ef89740 1 -- 172.21.15.123:0/3938182818 wait complete. 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 -- start start 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 Processor -- start 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 0x55f38c716610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c70ae10 0x55f38c715160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b63f0 con 0x55f38c7052b0 2024-03-24T00:42:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b7710 con 0x55f38c705ca0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.468+0000 7f769ef89740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941b6a50 con 0x55f38c70ae10 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f769ef89740 10 client.0 _ll_register_callbacks cb 0x7f76880b89e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59778/0 (socket says 172.21.15.123:59778) 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 -- 172.21.15.123:0/1079281485 learned_addr learned my addr 172.21.15.123:0/1079281485 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7699800640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 0x55f38c716610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f769a200640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c70ae10 0x55f38c715160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 -- 172.21.15.123:0/1079281485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c70ae10 msgr2=0x55f38c715160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c70ae10 0x55f38c715160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 -- 172.21.15.123:0/1079281485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 msgr2=0x55f38c716610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 0x55f38c716610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 -- 172.21.15.123:0/1079281485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f38c715820 con 0x55f38c705ca0 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7699800640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 0x55f38c716610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f7698e00640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f76880bc670 tx=0x7f768809d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c0d7e000362d79d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.469+0000 7f767b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880c7030 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f76880b83a0 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f768c01aa70 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768809f070 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f769ef89740 10 client.25997 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f38c7778c0 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f768c01aa70 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.470+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f76880b85a0 con 0x55f38c705ca0 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 10 client.25997 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 20 client.25997 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:34.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c778030 0x55f38c77a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 4 client.25997 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:42:34.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f38c672740 con 0x55f38c778030 2024-03-24T00:42:34.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.471+0000 7f7699800640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c778030 0x55f38c77a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.472+0000 7f7699800640 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c778030 0x55f38c77a450 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c835d972edda1219 server_cookie=8d8b65853f40c42d in_seq=0 out_seq=0 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.472+0000 7f767b400640 10 client.25997 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.473+0000 7f767b400640 1 -- 172.21.15.123:0/1079281485 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f38c672740 con 0x55f38c778030 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.473+0000 7f767b400640 10 client.25997 handle_command_reply: tid=0 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.473+0000 7f769ef89740 1 client.25997 shutdown 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 msgr2=0x55f38c714c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f76880bc670 tx=0x7f768809d000 comp rx=0 tx=0).stop 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 shutdown_connections 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f38c778030 0x55f38c77a450 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f38c70ae10 0x55f38c715160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f38c705ca0 0x55f38c714c20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 --2- 172.21.15.123:0/1079281485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f38c7052b0 0x55f38c716610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f767b400640 0 client.25997 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 >> 172.21.15.123:0/1079281485 conn(0x55f38c700820 msgr2=0x55f38c706fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 shutdown_connections 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 1 -- 172.21.15.123:0/1079281485 wait complete. 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.474+0000 7f769ef89740 20 client.25997 trim_cache size 0 max 16384 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "2ee85fda-a77b-43b3-9682-9ad2b5090b85", 2024-03-24T00:42:34.477 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:42:34.478 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:42:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 msgr2=0x7f767c0a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 0x7f767c0a4d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7694110760 tx=0x7f7680009260 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 msgr2=0x7f7694110f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7688009780 tx=0x7f7688004270 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 shutdown_connections 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f767c0a28c0 0x7f767c0a4d80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7694103150 0x7f7694111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7694102790 0x7f7694110f50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 --2- 172.21.15.123:0/1139026863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76941015a0 0x7f7694110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 >> 172.21.15.123:0/1139026863 conn(0x7f76940fcb10 msgr2=0x7f7694105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 shutdown_connections 2024-03-24T00:42:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.475+0000 7f769ac00640 1 -- 172.21.15.123:0/1139026863 wait complete. 2024-03-24T00:42:34.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:42:34.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:42:34.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 -- 172.21.15.123:0/4216100153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 msgr2=0x7fa8a0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 --2- 172.21.15.123:0/4216100153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0104e10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009900 tx=0x7fa88c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 -- 172.21.15.123:0/4216100153 shutdown_connections 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 --2- 172.21.15.123:0/4216100153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a0105350 0x7fa8a010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 --2- 172.21.15.123:0/4216100153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0104e10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 --2- 172.21.15.123:0/4216100153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a01037a0 0x7fa8a0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 -- 172.21.15.123:0/4216100153 >> 172.21.15.123:0/4216100153 conn(0x7fa8a00fed10 msgr2=0x7fa8a0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 -- 172.21.15.123:0/4216100153 shutdown_connections 2024-03-24T00:42:34.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.701+0000 7fa8a5600640 1 -- 172.21.15.123:0/4216100153 wait complete. 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 Processor -- start 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 -- start start 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 0x7fa8a01a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a010f470 con 0x7fa8a0105350 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a010f2f0 con 0x7fa8a01037a0 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa8a5600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a010f5f0 con 0x7fa8a0104990 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59800/0 (socket says 172.21.15.123:59800) 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.703+0000 7fa89fe00640 1 -- 172.21.15.123:0/939445313 learned_addr learned my addr 172.21.15.123:0/939445313 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 0x7fa8a01a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89f400640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 -- 172.21.15.123:0/939445313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 msgr2=0x7fa8a0199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 -- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 msgr2=0x7fa8a01a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 0x7fa8a01a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 -- 172.21.15.123:0/939445313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a01a16d0 con 0x7fa8a01037a0 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 0x7fa8a01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:34.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89fe00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa89000e9c0 tx=0x7fa89000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89cc00640 1 -- 172.21.15.123:0/939445313 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000cd80 con 0x7fa8a01037a0 2024-03-24T00:42:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89cc00640 1 -- 172.21.15.123:0/939445313 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8900035f0 con 0x7fa8a01037a0 2024-03-24T00:42:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa89cc00640 1 -- 172.21.15.123:0/939445313 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890010690 con 0x7fa8a01037a0 2024-03-24T00:42:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.704+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a01a19c0 con 0x7fa8a01037a0 2024-03-24T00:42:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.705+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a01a1e90 con 0x7fa8a01037a0 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 Processor -- start 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa89cc00640 1 -- 172.21.15.123:0/939445313 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa8900037a0 con 0x7fa8a01037a0 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 -- start start 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf82470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf8a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabcee7250 con 0x55eabcf82050 2024-03-24T00:42:34.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.708+0000 7fa8a982e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabce8fa30 con 0x55eabcf82a40 2024-03-24T00:42:34.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa8a982e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabcee32f0 con 0x55eabcf87bb0 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf8a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59804/0 (socket says 172.21.15.123:59804) 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 -- 172.21.15.123:0/3118532393 learned_addr learned my addr 172.21.15.123:0/3118532393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf82470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 -- 172.21.15.123:0/3118532393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 msgr2=0x55eabcf8a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf8a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 -- 172.21.15.123:0/3118532393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 msgr2=0x55eabcf82470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf82470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa89fe00640 1 -- 172.21.15.123:0/3118532393 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eabce7a690 con 0x55eabcf82a40 2024-03-24T00:42:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.709+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf82470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa89fe00640 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa8900d2da0 tx=0x7fa8900d74b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83225cacfd5da310 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa895200640 1 -- 172.21.15.123:0/3118532393 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900e2070 con 0x55eabcf82a40 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa895200640 1 -- 172.21.15.123:0/3118532393 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8900dd8e0 con 0x55eabcf82a40 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa895200640 1 -- 172.21.15.123:0/3118532393 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900dca10 con 0x55eabcf82a40 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa89cc00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 0x7fa8840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa89cc00640 1 -- 172.21.15.123:0/939445313 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa8900cee90 con 0x7fa8a01037a0 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.710+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 0x7fa8840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- 172.21.15.123:0/3118532393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 msgr2=0x55eabcf87540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa8900d2da0 tx=0x7fa8900d74b0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- 172.21.15.123:0/3118532393 shutdown_connections 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf8a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf87540 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 --2- 172.21.15.123:0/3118532393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf82470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- 172.21.15.123:0/3118532393 >> 172.21.15.123:0/3118532393 conn(0x55eabcf4a200 msgr2=0x55eabcf7d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- 172.21.15.123:0/3118532393 shutdown_connections 2024-03-24T00:42:34.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- 172.21.15.123:0/3118532393 wait complete. 2024-03-24T00:42:34.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 0x7fa8840a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55eabcf832a0 tx=0x7fa898009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.711+0000 7fa8a982e740 1 -- start start 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 Processor -- start 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82050 0x55eabcf8c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf87bb0 0x55eabcf935d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabcf8ab60 con 0x55eabcf87bb0 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabcea2e80 con 0x55eabcf82a40 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eabcf8ace0 con 0x55eabcf82050 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a982e740 10 client.0 _ll_register_callbacks cb 0x55eabceb5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa89f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa89f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59820/0 (socket says 172.21.15.123:59820) 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa89f400640 1 -- 172.21.15.123:0/117641422 learned_addr learned my addr 172.21.15.123:0/117641422 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.712+0000 7fa8a4c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf87bb0 0x55eabcf935d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89fe00640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82050 0x55eabcf8c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 -- 172.21.15.123:0/117641422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82050 msgr2=0x55eabcf8c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82050 0x55eabcf8c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 -- 172.21.15.123:0/117641422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf87bb0 msgr2=0x55eabcf935d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf87bb0 0x55eabcf935d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 -- 172.21.15.123:0/117641422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eabccc57d0 con 0x55eabcf82a40 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa89f400640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa88c005ab0 tx=0x7fa88c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d0695b5c29058d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa889600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.713+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c005d50 con 0x55eabcf82a40 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa88c0046c0 con 0x55eabcf82a40 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8900dd8e0 con 0x55eabcf82a40 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa88c0048c0 con 0x55eabcf82a40 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 10 client.26009 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:34.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55eabcbfb3f0 con 0x55eabcf82a40 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8900dd8e0 con 0x55eabcf82a40 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa88c0036d0 con 0x55eabcf82a40 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 10 client.26009 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 10 client.26009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 20 client.26009 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcf97e80 0x55eabcf9a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 4 client.26009 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.714+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eabcef4790 con 0x55eabcf97e80 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.715+0000 7fa89fe00640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcf97e80 0x55eabcf9a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.715+0000 7fa89fe00640 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcf97e80 0x55eabcf9a2c0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ca040e8278ad042 server_cookie=52e7ef47a67e1cce in_seq=0 out_seq=0 2024-03-24T00:42:34.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.715+0000 7fa889600640 10 client.26009 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa889600640 1 -- 172.21.15.123:0/117641422 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eabcef4790 con 0x55eabcf97e80 2024-03-24T00:42:34.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa889600640 10 client.26009 handle_command_reply: tid=0 2024-03-24T00:42:34.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 client.26009 shutdown 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 msgr2=0x55eabcf8c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa88c005ab0 tx=0x7fa88c004270 comp rx=0 tx=0).stop 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 shutdown_connections 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf87bb0 0x55eabcf935d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82a40 0x55eabcf8c590 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcf97e80 0x55eabcf9a2c0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 --2- 172.21.15.123:0/117641422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82050 0x55eabcf8c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa889600640 0 client.26009 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.716+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 >> 172.21.15.123:0/117641422 conn(0x55eabcf4a200 msgr2=0x55eabcf7cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.717+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 shutdown_connections 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.717+0000 7fa8a982e740 1 -- 172.21.15.123:0/117641422 wait complete. 2024-03-24T00:42:34.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.717+0000 7fa8a982e740 20 client.26009 trim_cache size 0 max 16384 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.736+0000 7fa8a982e740 1 Processor -- start 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.736+0000 7fa8a982e740 1 -- start start 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 0x55eabcf78920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82a40 0x55eabcf93590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a8be0 con 0x55eabcf82a40 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01ab1a0 con 0x55eabcf82050 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a982e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a91e0 con 0x55eabcf87bb0 2024-03-24T00:42:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 0x55eabcf78920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 0x55eabcf78920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59826/0 (socket says 172.21.15.123:59826) 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa89f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82a40 0x55eabcf93590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa89fe00640 1 -- 172.21.15.123:0/4284911411 learned_addr learned my addr 172.21.15.123:0/4284911411 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 -- 172.21.15.123:0/4284911411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 msgr2=0x55eabcf78920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 0x55eabcf78920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 -- 172.21.15.123:0/4284911411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82a40 msgr2=0x55eabcf93590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82a40 0x55eabcf93590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.737+0000 7fa8a4c00640 1 -- 172.21.15.123:0/4284911411 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eabcbbb0f0 con 0x55eabcf87bb0 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.738+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf81310 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa898013fd0 tx=0x7fa898019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65373d050632e283 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.738+0000 7fa88be00640 1 -- 172.21.15.123:0/4284911411 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898004d20 con 0x55eabcf87bb0 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.738+0000 7fa88be00640 1 -- 172.21.15.123:0/4284911411 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa89801cbe0 con 0x55eabcf87bb0 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.738+0000 7fa88be00640 1 -- 172.21.15.123:0/4284911411 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8980058e0 con 0x55eabcf87bb0 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- 172.21.15.123:0/4284911411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 msgr2=0x55eabcf81310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf81310 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa898013fd0 tx=0x7fa898019000 comp rx=0 tx=0).stop 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- 172.21.15.123:0/4284911411 shutdown_connections 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf87bb0 0x55eabcf81310 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82a40 0x55eabcf93590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 --2- 172.21.15.123:0/4284911411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf82050 0x55eabcf78920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- 172.21.15.123:0/4284911411 >> 172.21.15.123:0/4284911411 conn(0x55eabcf4a200 msgr2=0x55eabcf80c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- 172.21.15.123:0/4284911411 shutdown_connections 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- 172.21.15.123:0/4284911411 wait complete. 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.739+0000 7fa8a982e740 1 -- start start 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 Processor -- start 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf93380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82a40 0x55eabcf91990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a9820 con 0x55eabcf82050 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01aab40 con 0x55eabcf87bb0 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a9e80 con 0x55eabcf82a40 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a982e740 10 client.0 _ll_register_callbacks cb 0x7fa88c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf93380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa89f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82a40 0x55eabcf91990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:59830/0 (socket says 172.21.15.123:59830) 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.740+0000 7fa8a4c00640 1 -- 172.21.15.123:0/2092016080 learned_addr learned my addr 172.21.15.123:0/2092016080 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 -- 172.21.15.123:0/2092016080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82a40 msgr2=0x55eabcf91990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82a40 0x55eabcf91990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 -- 172.21.15.123:0/2092016080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 msgr2=0x55eabcf93380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf93380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 -- 172.21.15.123:0/2092016080 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eabcf925e0 con 0x55eabcf87bb0 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a4c00640 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa898010620 tx=0x7fa898005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=337691cb8ce1ff69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa889600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898036020 con 0x55eabcf87bb0 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa898004a40 con 0x55eabcf87bb0 2024-03-24T00:42:34.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.741+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa89801cbe0 con 0x55eabcf87bb0 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa898004c40 con 0x55eabcf87bb0 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 10 client.26015 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55eabcff46a0 con 0x55eabcf87bb0 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa89801cbe0 con 0x55eabcf87bb0 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa898027070 con 0x55eabcf87bb0 2024-03-24T00:42:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 10 client.26015 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 10 client.26015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 20 client.26015 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcff5030 0x55eabcff7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 4 client.26015 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eabceeeae0 con 0x55eabcff5030 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.742+0000 7fa89fe00640 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcff5030 0x55eabcff7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa89fe00640 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcff5030 0x55eabcff7450 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1305aa756616265 server_cookie=378ea2500c4a9769 in_seq=0 out_seq=0 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa889600640 10 client.26015 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa889600640 1 -- 172.21.15.123:0/2092016080 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55eabceeeae0 con 0x55eabcff5030 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa889600640 10 client.26015 handle_command_reply: tid=0 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 client.26015 shutdown 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 msgr2=0x55eabcf91ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa898010620 tx=0x7fa898005280 comp rx=0 tx=0).stop 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 shutdown_connections 2024-03-24T00:42:34.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eabcf87bb0 0x55eabcf91ed0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55eabcff5030 0x55eabcff7450 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eabcf82a40 0x55eabcf91990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 --2- 172.21.15.123:0/2092016080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55eabcf82050 0x55eabcf93380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 >> 172.21.15.123:0/2092016080 conn(0x55eabcf4a200 msgr2=0x55eabcf7cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa889600640 0 client.26015 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 shutdown_connections 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.743+0000 7fa8a982e740 1 -- 172.21.15.123:0/2092016080 wait complete. 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a982e740 20 client.26015 trim_cache size 0 max 16384 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (70 inodes in the stack)", 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "2ee85fda-a77b-43b3-9682-9ad2b5090b85": { 2024-03-24T00:42:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "2ee85fda-a77b-43b3-9682-9ad2b5090b85", 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:42:34.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 msgr2=0x7fa8840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 0x7fa8840a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55eabcf832a0 tx=0x7fa898009500 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 msgr2=0x7fa8a0199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa89000e9c0 tx=0x7fa89000ee80 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 shutdown_connections 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa8840a2880 0x7fa8840a4d40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8a0105350 0x7fa8a01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8a0104990 0x7fa8a0199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 --2- 172.21.15.123:0/939445313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8a01037a0 0x7fa8a0199ab0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.744+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 >> 172.21.15.123:0/939445313 conn(0x7fa8a00fed10 msgr2=0x7fa8a0107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.745+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 shutdown_connections 2024-03-24T00:42:34.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:42:34.745+0000 7fa8a5600640 1 -- 172.21.15.123:0/939445313 wait complete. 2024-03-24T00:42:34.765 INFO:tasks.cephfs.filesystem:scrub status for tag:2ee85fda-a77b-43b3-9682-9ad2b5090b85 - {'path': '/', 'tag': '2ee85fda-a77b-43b3-9682-9ad2b5090b85', 'options': 'recursive,force'} 2024-03-24T00:42:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:35 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:42:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:35 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:35.462 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:35 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:42:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:35 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:42:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:35 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:35.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:35 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:42:35.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:35 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:42:35.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:35 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:35.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:35 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:42:36.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:36 smithi177 ceph-mon[36034]: pgmap v526: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 71 MiB/s wr, 6.60k op/s 2024-03-24T00:42:36.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:36 smithi204 ceph-mon[36345]: pgmap v526: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 71 MiB/s wr, 6.60k op/s 2024-03-24T00:42:36.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:36 smithi123 ceph-mon[31487]: pgmap v526: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 71 MiB/s wr, 6.60k op/s 2024-03-24T00:42:38.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:38 smithi204 ceph-mon[36345]: pgmap v527: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 64 MiB/s wr, 5.94k op/s 2024-03-24T00:42:38.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:38 smithi123 ceph-mon[31487]: pgmap v527: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 64 MiB/s wr, 5.94k op/s 2024-03-24T00:42:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:38 smithi177 ceph-mon[36034]: pgmap v527: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 64 MiB/s wr, 5.94k op/s 2024-03-24T00:42:39.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:39 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:39.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:39 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:39.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:39 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:42:40.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:40 smithi123 ceph-mon[31487]: pgmap v528: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 65 MiB/s wr, 6.17k op/s 2024-03-24T00:42:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:40 smithi177 ceph-mon[36034]: pgmap v528: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 65 MiB/s wr, 6.17k op/s 2024-03-24T00:42:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:40 smithi204 ceph-mon[36345]: pgmap v528: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 65 MiB/s wr, 6.17k op/s 2024-03-24T00:42:42.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:42 smithi123 ceph-mon[31487]: pgmap v529: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 67 MiB/s wr, 6.37k op/s 2024-03-24T00:42:42.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:42 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:42:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:42 smithi177 ceph-mon[36034]: pgmap v529: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 67 MiB/s wr, 6.37k op/s 2024-03-24T00:42:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:42 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:42:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:42 smithi204 ceph-mon[36345]: pgmap v529: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 67 MiB/s wr, 6.37k op/s 2024-03-24T00:42:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:42 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:42:44.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:44 smithi123 ceph-mon[31487]: pgmap v530: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 62 MiB/s wr, 5.99k op/s 2024-03-24T00:42:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:44 smithi177 ceph-mon[36034]: pgmap v530: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 62 MiB/s wr, 5.99k op/s 2024-03-24T00:42:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:44 smithi204 ceph-mon[36345]: pgmap v530: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 62 MiB/s wr, 5.99k op/s 2024-03-24T00:42:46.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:46 smithi123 ceph-mon[31487]: pgmap v531: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 59 MiB/s wr, 5.62k op/s 2024-03-24T00:42:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:46 smithi177 ceph-mon[36034]: pgmap v531: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 59 MiB/s wr, 5.62k op/s 2024-03-24T00:42:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:46 smithi204 ceph-mon[36345]: pgmap v531: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 59 MiB/s wr, 5.62k op/s 2024-03-24T00:42:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.670+0000 7f4e51c00640 1 -- 172.21.15.123:0/3299962703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 msgr2=0x7f4e4c10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.670+0000 7f4e51c00640 1 --2- 172.21.15.123:0/3299962703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c10d6e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009900 tx=0x7f4e3802bc30 comp rx=0 tx=0).stop 2024-03-24T00:42:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.673+0000 7f4e51c00640 1 -- 172.21.15.123:0/3299962703 shutdown_connections 2024-03-24T00:42:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.673+0000 7f4e51c00640 1 --2- 172.21.15.123:0/3299962703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.673+0000 7f4e51c00640 1 --2- 172.21.15.123:0/3299962703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c10d6e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.673+0000 7f4e51c00640 1 --2- 172.21.15.123:0/3299962703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c10ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.674+0000 7f4e51c00640 1 -- 172.21.15.123:0/3299962703 >> 172.21.15.123:0/3299962703 conn(0x7f4e4c06e6e0 msgr2=0x7f4e4c070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.674+0000 7f4e51c00640 1 -- 172.21.15.123:0/3299962703 shutdown_connections 2024-03-24T00:42:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.674+0000 7f4e51c00640 1 -- 172.21.15.123:0/3299962703 wait complete. 2024-03-24T00:42:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.676+0000 7f4e51c00640 1 Processor -- start 2024-03-24T00:42:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.676+0000 7f4e51c00640 1 -- start start 2024-03-24T00:42:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.677+0000 7f4e51c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c1a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.677+0000 7f4e51c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c1a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.677+0000 7f4e51c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.677+0000 7f4e4a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c1a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e4a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c1a6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:60532/0 (socket says 172.21.15.123:60532) 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:57618/0 (socket says 172.21.15.123:57618) 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e51c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c113590 con 0x7f4e4c108910 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e51c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c113410 con 0x7f4e4c10dd70 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e51c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c113710 con 0x7f4e4c10b2d0 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.678+0000 7f4e4a000640 1 -- 172.21.15.123:0/2357567488 learned_addr learned my addr 172.21.15.123:0/2357567488 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c1a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4b400640 1 -- 172.21.15.123:0/2357567488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 msgr2=0x7f4e4c1a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4b400640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c1a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4b400640 1 -- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 msgr2=0x7f4e4c1a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4b400640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c1a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4b400640 1 -- 172.21.15.123:0/2357567488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c1ae310 con 0x7f4e4c10dd70 2024-03-24T00:42:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.679+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c1a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.680+0000 7f4e4b400640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00e970 tx=0x7f4e3c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.680+0000 7f4e43400640 1 -- 172.21.15.123:0/2357567488 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00ccc0 con 0x7f4e4c10dd70 2024-03-24T00:42:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.681+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c1ae600 con 0x7f4e4c10dd70 2024-03-24T00:42:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.681+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c1aeb70 con 0x7f4e4c10dd70 2024-03-24T00:42:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.681+0000 7f4e43400640 1 -- 172.21.15.123:0/2357567488 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3c005020 con 0x7f4e4c10dd70 2024-03-24T00:42:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.682+0000 7f4e43400640 1 -- 172.21.15.123:0/2357567488 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00ccc0 con 0x7f4e4c10dd70 2024-03-24T00:42:47.692 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.691+0000 7f4e54831740 1 Processor -- start 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.692+0000 7f4e43400640 1 -- 172.21.15.123:0/2357567488 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4e3c0047e0 con 0x7f4e4c10dd70 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 -- start start 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 0x560345c84600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 0x560345c896d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c89d40 0x560345c8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345c473b0 con 0x560345c89d40 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345be9830 con 0x560345c841e0 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.694+0000 7f4e54831740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345b925d0 con 0x560345c84bd0 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.695+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 0x560345c896d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.695+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 0x560345c896d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:60536/0 (socket says 172.21.15.123:60536) 2024-03-24T00:42:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.695+0000 7f4e4aa00640 1 -- 172.21.15.123:0/2381347271 learned_addr learned my addr 172.21.15.123:0/2381347271 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.695+0000 7f4e4a000640 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 0x560345c84600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.695+0000 7f4e4a000640 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c89d40 0x560345c8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.697+0000 7f4e4a000640 1 -- 172.21.15.123:0/2381347271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 msgr2=0x560345c896d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.697+0000 7f4e4a000640 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 0x560345c896d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.697+0000 7f4e4a000640 1 -- 172.21.15.123:0/2381347271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c89d40 msgr2=0x560345c8c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.697+0000 7f4e4a000640 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c89d40 0x560345c8c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.697+0000 7f4e4a000640 1 -- 172.21.15.123:0/2381347271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560345baa2c0 con 0x560345c841e0 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.698+0000 7f4e43400640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 0x7f4e300a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.698+0000 7f4e43400640 1 -- 172.21.15.123:0/2357567488 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4e3c0ce5f0 con 0x7f4e4c10dd70 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.700+0000 7f4e4a000640 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 0x560345c84600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4e38000c00 tx=0x7f4e38005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ead4e18ad0f6d48c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.700+0000 7f4e37e00640 1 -- 172.21.15.123:0/2381347271 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38004030 con 0x560345c841e0 2024-03-24T00:42:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.700+0000 7f4e37e00640 1 -- 172.21.15.123:0/2381347271 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e38004ba0 con 0x560345c841e0 2024-03-24T00:42:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.700+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 0x7f4e300a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.701+0000 7f4e54831740 1 -- 172.21.15.123:0/2381347271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 msgr2=0x560345c84600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.701+0000 7f4e37e00640 1 -- 172.21.15.123:0/2381347271 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38004030 con 0x560345c841e0 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.701+0000 7f4e54831740 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 0x560345c84600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4e38000c00 tx=0x7f4e38005b50 comp rx=0 tx=0).stop 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.702+0000 7f4e54831740 1 -- 172.21.15.123:0/2381347271 shutdown_connections 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.702+0000 7f4e54831740 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c89d40 0x560345c8c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.702+0000 7f4e54831740 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c84bd0 0x560345c896d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.702+0000 7f4e54831740 1 --2- 172.21.15.123:0/2381347271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c841e0 0x560345c84600 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.702+0000 7f4e54831740 1 -- 172.21.15.123:0/2381347271 >> 172.21.15.123:0/2381347271 conn(0x560345c7f750 msgr2=0x560345c7fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.704+0000 7f4e54831740 1 -- 172.21.15.123:0/2381347271 shutdown_connections 2024-03-24T00:42:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.704+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 0x7f4e300a4ff0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x560345c84fd0 tx=0x7f4e44009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.704+0000 7f4e54831740 1 -- 172.21.15.123:0/2381347271 wait complete. 2024-03-24T00:42:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.705+0000 7f4e54831740 1 -- start start 2024-03-24T00:42:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.705+0000 7f4e54831740 1 Processor -- start 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.706+0000 7f4e54831740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c8e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.706+0000 7f4e54831740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c8e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e54831740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c8e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c8e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:60546/0 (socket says 172.21.15.123:60546) 2024-03-24T00:42:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:57640/0 (socket says 172.21.15.123:57640) 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.707+0000 7f4e4b400640 1 -- 172.21.15.123:0/1397066033 learned_addr learned my addr 172.21.15.123:0/1397066033 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.708+0000 7f4e4a000640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c8e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.708+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345ba5a20 con 0x560345c84bd0 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.708+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345be58d0 con 0x560345c89d40 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.708+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560345c8cce0 con 0x560345c841e0 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.708+0000 7f4e54831740 10 client.0 _ll_register_callbacks cb 0x560345be21c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.709+0000 7f4e4b400640 1 -- 172.21.15.123:0/1397066033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 msgr2=0x560345c8e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.709+0000 7f4e4b400640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c8e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.709+0000 7f4e4b400640 1 -- 172.21.15.123:0/1397066033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 msgr2=0x560345c8e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.710+0000 7f4e4b400640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c8e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.710+0000 7f4e4b400640 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603458fe210 con 0x560345c89d40 2024-03-24T00:42:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.710+0000 7f4e4a000640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c8e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.711+0000 7f4e34c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:42:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.711+0000 7f4e4b400640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c09bf10 tx=0x7f4e3c09c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fa5b072ef3142f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.712+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c01c040 con 0x560345c89d40 2024-03-24T00:42:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.712+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e38004ba0 con 0x560345c89d40 2024-03-24T00:42:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.712+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4e3c0a4290 con 0x560345c89d40 2024-03-24T00:42:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.713+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e3c01c040 con 0x560345c89d40 2024-03-24T00:42:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.713+0000 7f4e54831740 10 client.26027 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.713+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5603459ca620 con 0x560345c89d40 2024-03-24T00:42:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.714+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e38004ba0 con 0x560345c89d40 2024-03-24T00:42:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.714+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4e3c09b3d0 con 0x560345c89d40 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 10 client.26027 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 10 client.26027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 20 client.26027 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c99fd0 0x560345c9c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 4 client.26027 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.716+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560345bf6d70 con 0x560345c99fd0 2024-03-24T00:42:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.717+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c99fd0 0x560345c9c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.720+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c99fd0 0x560345c9c420 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c395aa871f89b922 server_cookie=bed7b5520c26724c in_seq=0 out_seq=0 2024-03-24T00:42:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.720+0000 7f4e34c00640 10 client.26027 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.721+0000 7f4e34c00640 1 -- 172.21.15.123:0/1397066033 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560345bf6d70 con 0x560345c99fd0 2024-03-24T00:42:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.721+0000 7f4e34c00640 10 client.26027 handle_command_reply: tid=0 2024-03-24T00:42:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.721+0000 7f4e54831740 1 client.26027 shutdown 2024-03-24T00:42:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.723+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 msgr2=0x560345c95920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.723+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c09bf10 tx=0x7f4e3c09c7c0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 shutdown_connections 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c89d40 0x560345c95920 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c8e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c99fd0 0x560345c9c420 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 --2- 172.21.15.123:0/1397066033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c8e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e34c00640 0 client.26027 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.724+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 >> 172.21.15.123:0/1397066033 conn(0x560345c7f750 msgr2=0x560345c7e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.725+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 shutdown_connections 2024-03-24T00:42:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.725+0000 7f4e54831740 1 -- 172.21.15.123:0/1397066033 wait complete. 2024-03-24T00:42:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.725+0000 7f4e54831740 20 client.26027 trim_cache size 0 max 16384 2024-03-24T00:42:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.760+0000 7f4e54831740 1 Processor -- start 2024-03-24T00:42:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.760+0000 7f4e54831740 1 -- start start 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00c3c0 con 0x560345c84bd0 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00e9e0 con 0x560345c7f750 2024-03-24T00:42:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.761+0000 7f4e54831740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00ca00 con 0x560345c841e0 2024-03-24T00:42:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.762+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.762+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:57652/0 (socket says 172.21.15.123:57652) 2024-03-24T00:42:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.762+0000 7f4e4aa00640 1 -- 172.21.15.123:0/889546608 learned_addr learned my addr 172.21.15.123:0/889546608 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.762+0000 7f4e4b400640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.762+0000 7f4e4a000640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 -- 172.21.15.123:0/889546608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 msgr2=0x560345c958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 -- 172.21.15.123:0/889546608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 msgr2=0x560345c830c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c830c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 -- 172.21.15.123:0/889546608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560345c83e20 con 0x560345c7f750 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4a000640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.763+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x560345c806c0 tx=0x7f4e44004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed80aa7f7e744d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.764+0000 7f4e37400640 1 -- 172.21.15.123:0/889546608 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4402d070 con 0x560345c7f750 2024-03-24T00:42:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.764+0000 7f4e37400640 1 -- 172.21.15.123:0/889546608 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e44027d10 con 0x560345c7f750 2024-03-24T00:42:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.764+0000 7f4e37400640 1 -- 172.21.15.123:0/889546608 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440042a0 con 0x560345c7f750 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.765+0000 7f4e54831740 1 -- 172.21.15.123:0/889546608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 msgr2=0x560345c7a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x560345c806c0 tx=0x7f4e44004a40 comp rx=0 tx=0).stop 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 -- 172.21.15.123:0/889546608 shutdown_connections 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c830c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 --2- 172.21.15.123:0/889546608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c7a3a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 -- 172.21.15.123:0/889546608 >> 172.21.15.123:0/889546608 conn(0x560345c89840 msgr2=0x560345c829b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.766+0000 7f4e54831740 1 -- 172.21.15.123:0/889546608 shutdown_connections 2024-03-24T00:42:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.767+0000 7f4e54831740 1 -- 172.21.15.123:0/889546608 wait complete. 2024-03-24T00:42:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.768+0000 7f4e54831740 1 -- start start 2024-03-24T00:42:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.768+0000 7f4e54831740 1 Processor -- start 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c92d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c932b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c94220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00d060 con 0x560345c84bd0 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00e380 con 0x560345c7f750 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00d6c0 con 0x560345c841e0 2024-03-24T00:42:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.769+0000 7f4e54831740 10 client.0 _ll_register_callbacks cb 0x7f4e3c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.770+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c94220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.770+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c94220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39632/0 (socket says 172.21.15.123:39632) 2024-03-24T00:42:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.770+0000 7f4e4b400640 1 -- 172.21.15.123:0/3794312445 learned_addr learned my addr 172.21.15.123:0/3794312445 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:42:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.770+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c92d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.770+0000 7f4e4a000640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c932b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 -- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 msgr2=0x560345c92d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c92d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 -- 172.21.15.123:0/3794312445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 msgr2=0x560345c94220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c94220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 -- 172.21.15.123:0/3794312445 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560345c94810 con 0x560345c841e0 2024-03-24T00:42:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.771+0000 7f4e4a000640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c932b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4e3802c240 tx=0x7f4e380045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa346b580617cb37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.772+0000 7f4e34c00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:42:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.772+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38051070 con 0x560345c841e0 2024-03-24T00:42:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.772+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4e38004ad0 con 0x560345c841e0 2024-03-24T00:42:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.772+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e44027d10 con 0x560345c841e0 2024-03-24T00:42:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.773+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e38004f60 con 0x560345c841e0 2024-03-24T00:42:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.773+0000 7f4e54831740 10 client.26038 fetch_fsmap learned FSMap version 19 2024-03-24T00:42:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.773+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560345c95b20 con 0x560345c841e0 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e44027d10 con 0x560345c841e0 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4e380038e0 con 0x560345c841e0 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 10 client.26038 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 10 client.26038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 20 client.26038 populate_metadata read hostname 'smithi123' 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c89d40 0x560345cf7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 4 client.26038 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:42:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560345c7aa80 con 0x560345c89d40 2024-03-24T00:42:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.774+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c89d40 0x560345cf7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.775+0000 7f4e34c00640 10 client.26038 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:42:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:42:47.775+0000 7f4e4aa00640 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c89d40 0x560345cf7b10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b3944dd8ccd9ae1 server_cookie=93b7f66a56481d5a in_seq=0 out_seq=0 2024-03-24T00:42:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:48 smithi177 ceph-mon[36034]: pgmap v532: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 49 MiB/s wr, 4.45k op/s 2024-03-24T00:42:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:48 smithi204 ceph-mon[36345]: pgmap v532: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 49 MiB/s wr, 4.45k op/s 2024-03-24T00:42:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:48 smithi123 ceph-mon[31487]: pgmap v532: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 49 MiB/s wr, 4.45k op/s 2024-03-24T00:42:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:50 smithi177 ceph-mon[36034]: pgmap v533: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 57 MiB/s wr, 5.28k op/s 2024-03-24T00:42:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:50 smithi204 ceph-mon[36345]: pgmap v533: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 57 MiB/s wr, 5.28k op/s 2024-03-24T00:42:50.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:50 smithi123 ceph-mon[31487]: pgmap v533: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 57 MiB/s wr, 5.28k op/s 2024-03-24T00:42:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:52 smithi177 ceph-mon[36034]: pgmap v534: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 66 MiB/s wr, 6.03k op/s 2024-03-24T00:42:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:52 smithi204 ceph-mon[36345]: pgmap v534: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 66 MiB/s wr, 6.03k op/s 2024-03-24T00:42:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:52 smithi123 ceph-mon[31487]: pgmap v534: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 66 MiB/s wr, 6.03k op/s 2024-03-24T00:42:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:54 smithi177 ceph-mon[36034]: pgmap v535: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 61 MiB/s wr, 5.69k op/s 2024-03-24T00:42:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:54 smithi204 ceph-mon[36345]: pgmap v535: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 61 MiB/s wr, 5.69k op/s 2024-03-24T00:42:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:54 smithi123 ceph-mon[31487]: pgmap v535: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 61 MiB/s wr, 5.69k op/s 2024-03-24T00:42:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:56 smithi177 ceph-mon[36034]: pgmap v536: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 59 MiB/s wr, 5.57k op/s 2024-03-24T00:42:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:56 smithi204 ceph-mon[36345]: pgmap v536: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 59 MiB/s wr, 5.57k op/s 2024-03-24T00:42:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:56 smithi123 ceph-mon[31487]: pgmap v536: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 59 MiB/s wr, 5.57k op/s 2024-03-24T00:42:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:42:58 smithi177 ceph-mon[36034]: pgmap v537: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 51 MiB/s wr, 4.75k op/s 2024-03-24T00:42:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:42:58 smithi204 ceph-mon[36345]: pgmap v537: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 51 MiB/s wr, 4.75k op/s 2024-03-24T00:42:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:42:58 smithi123 ceph-mon[31487]: pgmap v537: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 51 MiB/s wr, 4.75k op/s 2024-03-24T00:43:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:00 smithi177 ceph-mon[36034]: pgmap v538: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 57 MiB/s wr, 5.31k op/s 2024-03-24T00:43:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:00 smithi204 ceph-mon[36345]: pgmap v538: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 57 MiB/s wr, 5.31k op/s 2024-03-24T00:43:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:00 smithi123 ceph-mon[31487]: pgmap v538: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 57 MiB/s wr, 5.31k op/s 2024-03-24T00:43:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:02 smithi177 ceph-mon[36034]: pgmap v539: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 59 MiB/s wr, 5.44k op/s 2024-03-24T00:43:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:02 smithi204 ceph-mon[36345]: pgmap v539: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 59 MiB/s wr, 5.44k op/s 2024-03-24T00:43:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:02 smithi123 ceph-mon[31487]: pgmap v539: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 59 MiB/s wr, 5.44k op/s 2024-03-24T00:43:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:04 smithi177 ceph-mon[36034]: pgmap v540: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 52 MiB/s wr, 4.84k op/s 2024-03-24T00:43:04.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:43:04.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:43:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:04 smithi204 ceph-mon[36345]: pgmap v540: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 52 MiB/s wr, 4.84k op/s 2024-03-24T00:43:04.848 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:04 smithi123 ceph-mon[31487]: pgmap v540: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 52 MiB/s wr, 4.84k op/s 2024-03-24T00:43:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 -- 172.21.15.123:0/3284985599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd174105370 msgr2=0x7fd17410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 --2- 172.21.15.123:0/3284985599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd174105370 0x7fd17410bc00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009ca0 tx=0x7fd16c02c100 comp rx=0 tx=0).stop 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 -- 172.21.15.123:0/3284985599 shutdown_connections 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 --2- 172.21.15.123:0/3284985599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd174105370 0x7fd17410bc00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 --2- 172.21.15.123:0/3284985599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1741049b0 0x7fd174104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 --2- 172.21.15.123:0/3284985599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 -- 172.21.15.123:0/3284985599 >> 172.21.15.123:0/3284985599 conn(0x7fd1740fed50 msgr2=0x7fd174101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 -- 172.21.15.123:0/3284985599 shutdown_connections 2024-03-24T00:43:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.992+0000 7fd173200640 1 -- 172.21.15.123:0/3284985599 wait complete. 2024-03-24T00:43:04.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.993+0000 7fd173200640 1 Processor -- start 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.993+0000 7fd173200640 1 -- start start 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 0x7fd174199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd174105370 0x7fd1741a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17410f440 con 0x7fd174105370 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17410f2c0 con 0x7fd1741037c0 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd173200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17410f5c0 con 0x7fd1741049b0 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54636/0 (socket says 172.21.15.123:54636) 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171e00640 1 -- 172.21.15.123:0/1870834834 learned_addr learned my addr 172.21.15.123:0/1870834834 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 0x7fd174199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd172800640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd174105370 0x7fd1741a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171e00640 1 -- 172.21.15.123:0/1870834834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 msgr2=0x7fd174199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.994+0000 7fd171e00640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 0x7fd174199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd171e00640 1 -- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd174105370 msgr2=0x7fd1741a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd171e00640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd174105370 0x7fd1741a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd171e00640 1 -- 172.21.15.123:0/1870834834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1741a1670 con 0x7fd1741037c0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd171400640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 0x7fd174199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd171e00640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e9c0 tx=0x7fd16400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd16aa00640 1 -- 172.21.15.123:0/1870834834 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16400cd80 con 0x7fd1741037c0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd16aa00640 1 -- 172.21.15.123:0/1870834834 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1640035f0 con 0x7fd1741037c0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd16aa00640 1 -- 172.21.15.123:0/1870834834 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164010690 con 0x7fd1741037c0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1741a1960 con 0x7fd1741037c0 2024-03-24T00:43:04.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.995+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1741a1e60 con 0x7fd1741037c0 2024-03-24T00:43:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.998+0000 7fd16aa00640 1 -- 172.21.15.123:0/1870834834 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd1640037a0 con 0x7fd1741037c0 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 Processor -- start 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 -- start start 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 0x55cc84f25470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 0x55cc84f2a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84e8a250 con 0x55cc84f25050 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84e32a30 con 0x55cc84f25a40 2024-03-24T00:43:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84e862f0 con 0x55cc84f2abb0 2024-03-24T00:43:05.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 0x55cc84f2a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd172800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 0x55cc84f25470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd171400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd171400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43098/0 (socket says 172.21.15.123:43098) 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd172800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 0x55cc84f25470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40982/0 (socket says 172.21.15.123:40982) 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd171400640 1 -- 172.21.15.123:0/3335021919 learned_addr learned my addr 172.21.15.123:0/3335021919 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:04.999+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 0x55cc84f2a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54638/0 (socket says 172.21.15.123:54638) 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 -- 172.21.15.123:0/3335021919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 msgr2=0x55cc84f2a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 0x55cc84f2a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 -- 172.21.15.123:0/3335021919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 msgr2=0x55cc84f25470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 0x55cc84f25470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 -- 172.21.15.123:0/3335021919 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc84e1d690 con 0x55cc84f2abb0 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd16aa00640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 0x7fd1580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd16aa00640 1 -- 172.21.15.123:0/1870834834 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd1640ced20 con 0x7fd1741037c0 2024-03-24T00:43:05.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd172800640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 0x7fd1580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.000+0000 7fd171400640 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd1600114f0 tx=0x7fd1600119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2eb6315abc27006 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.001+0000 7fd15f400640 1 -- 172.21.15.123:0/3335021919 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16001e070 con 0x55cc84f2abb0 2024-03-24T00:43:05.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.001+0000 7fd15f400640 1 -- 172.21.15.123:0/3335021919 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1600035f0 con 0x55cc84f2abb0 2024-03-24T00:43:05.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.001+0000 7fd15f400640 1 -- 172.21.15.123:0/3335021919 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16000f820 con 0x55cc84f2abb0 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3335021919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 msgr2=0x55cc84f2d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd172800640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 0x7fd1580a4d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd16c005e30 tx=0x7fd16c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd1600114f0 tx=0x7fd1600119b0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3335021919 shutdown_connections 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f2abb0 0x55cc84f2d040 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25a40 0x55cc84f2a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3335021919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25050 0x55cc84f25470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3335021919 >> 172.21.15.123:0/3335021919 conn(0x55cc84eed200 msgr2=0x55cc84f209c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3335021919 shutdown_connections 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3335021919 wait complete. 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.002+0000 7fd17bb7b740 1 -- start start 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 Processor -- start 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f2f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f36570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84f2db60 con 0x55cc84f25a40 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84e45e80 con 0x55cc84f2abb0 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc84f2dce0 con 0x55cc84f25050 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd17bb7b740 10 client.0 _ll_register_callbacks cb 0x55cc84e58130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:05.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd171e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd171e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:43106/0 (socket says 172.21.15.123:43106) 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd172800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f36570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd171e00640 1 -- 172.21.15.123:0/2506089409 learned_addr learned my addr 172.21.15.123:0/2506089409 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.003+0000 7fd171400640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f2f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 -- 172.21.15.123:0/2506089409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 msgr2=0x55cc84f2eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 -- 172.21.15.123:0/2506089409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 msgr2=0x55cc84f2f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f2f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 -- 172.21.15.123:0/2506089409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc84c687d0 con 0x55cc84f2abb0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd171e00640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd172800640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f36570 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd16c02c640 tx=0x7fd16c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5870d6ff4ece95b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd153e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c051020 con 0x55cc84f2abb0 2024-03-24T00:43:05.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd16c003f80 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.004+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1600035f0 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd16c039040 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 10 client.26045 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cc84b9e3f0 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1600035f0 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd16c04d880 con 0x55cc84f2abb0 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 10 client.26045 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 20 client.26045 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f3ad90 0x55cc84f3d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 4 client.26045 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc84e97790 con 0x55cc84f3ad90 2024-03-24T00:43:05.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.005+0000 7fd171e00640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f3ad90 0x55cc84f3d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.006+0000 7fd171e00640 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f3ad90 0x55cc84f3d230 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=680cf65335cd5ff7 server_cookie=5adb33c56edc6b62 in_seq=0 out_seq=0 2024-03-24T00:43:05.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:05.006+0000 7fd153e00640 10 client.26045 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:06 smithi177 ceph-mon[36034]: pgmap v541: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 53 MiB/s wr, 4.94k op/s 2024-03-24T00:43:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:06 smithi204 ceph-mon[36345]: pgmap v541: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 53 MiB/s wr, 4.94k op/s 2024-03-24T00:43:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:06 smithi123 ceph-mon[31487]: pgmap v541: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 53 MiB/s wr, 4.94k op/s 2024-03-24T00:43:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.241+0000 7f4e34c00640 1 -- 172.21.15.123:0/3794312445 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560345c7aa80 con 0x560345c89d40 2024-03-24T00:43:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.241+0000 7f4e34c00640 10 client.26038 handle_command_reply: tid=0 2024-03-24T00:43:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.241+0000 7f4e54831740 1 client.26038 shutdown 2024-03-24T00:43:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 msgr2=0x560345c932b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c932b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4e3802c240 tx=0x7f4e380045e0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 shutdown_connections 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560345c84bd0 0x560345c94220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560345c89d40 0x560345cf7b10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560345c841e0 0x560345c932b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 --2- 172.21.15.123:0/3794312445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560345c7f750 0x560345c92d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 >> 172.21.15.123:0/3794312445 conn(0x560345c89840 msgr2=0x560345c83240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e34c00640 0 client.26038 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:07.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.242+0000 7fd153e00640 1 -- 172.21.15.123:0/2506089409 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cc84e97790 con 0x55cc84f3ad90 2024-03-24T00:43:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.242+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 shutdown_connections 2024-03-24T00:43:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.243+0000 7f4e54831740 1 -- 172.21.15.123:0/3794312445 wait complete. 2024-03-24T00:43:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.243+0000 7f4e54831740 20 client.26038 trim_cache size 0 max 16384 2024-03-24T00:43:07.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.242+0000 7fd153e00640 10 client.26045 handle_command_reply: tid=0 2024-03-24T00:43:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.243+0000 7fd17bb7b740 1 client.26045 shutdown 2024-03-24T00:43:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.243+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 msgr2=0x55cc84f36570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.243+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f36570 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd16c02c640 tx=0x7fd16c0095c0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:43:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:43:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:43:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:43:07.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 shutdown_connections 2024-03-24T00:43:07.246 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.244+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 msgr2=0x7f4e300a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.246 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.244+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 0x7f4e300a4ff0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x560345c84fd0 tx=0x7f4e44009040 comp rx=0 tx=0).stop 2024-03-24T00:43:07.246 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.244+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 msgr2=0x7f4e4c1adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.246 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.244+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00e970 tx=0x7f4e3c00ee30 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f36570 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f2f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f3ad90 0x55cc84f3d230 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2506089409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.244+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 >> 172.21.15.123:0/2506089409 conn(0x55cc84eed200 msgr2=0x55cc84f1fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 shutdown_connections 2024-03-24T00:43:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e4c10dd70 0x7f4e4c1adca0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e4c10b2d0 0x7f4e4c1a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e300a2b30 0x7f4e300a4ff0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 --2- 172.21.15.123:0/2357567488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c108910 0x7f4e4c1a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 >> 172.21.15.123:0/2357567488 conn(0x7f4e4c06e6e0 msgr2=0x7f4e4c10a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.245+0000 7fd153e00640 0 client.26045 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:07.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.245+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 shutdown_connections 2024-03-24T00:43:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 shutdown_connections 2024-03-24T00:43:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.245+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2506089409 wait complete. 2024-03-24T00:43:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:07.245+0000 7f4e51c00640 1 -- 172.21.15.123:0/2357567488 wait complete. 2024-03-24T00:43:07.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.246+0000 7fd17bb7b740 20 client.26045 trim_cache size 0 max 16384 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.266+0000 7fd17bb7b740 1 Processor -- start 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.266+0000 7fd17bb7b740 1 -- start start 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.266+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.266+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25a40 0x55cc84f36530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f2abb0 0x55cc84f23c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741a8aa0 con 0x55cc84f2abb0 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741ab060 con 0x55cc84f25050 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741a90a0 con 0x55cc84f25a40 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38010/0 (socket says 172.21.15.123:38010) 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 -- 172.21.15.123:0/2153766267 learned_addr learned my addr 172.21.15.123:0/2153766267 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171400640 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25a40 0x55cc84f36530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd172800640 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f2abb0 0x55cc84f23c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 -- 172.21.15.123:0/2153766267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25a40 msgr2=0x55cc84f36530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25a40 0x55cc84f36530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 -- 172.21.15.123:0/2153766267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f2abb0 msgr2=0x55cc84f23c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f2abb0 0x55cc84f23c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.267+0000 7fd171e00640 1 -- 172.21.15.123:0/2153766267 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc84b5e0f0 con 0x55cc84f25050 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.268+0000 7fd171e00640 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd16409b660 tx=0x7fd1640d1200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba3cf24b05ae23c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.268+0000 7fd15ea00640 1 -- 172.21.15.123:0/2153766267 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640b4070 con 0x55cc84f25050 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.268+0000 7fd15ea00640 1 -- 172.21.15.123:0/2153766267 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd16409bdd0 con 0x55cc84f25050 2024-03-24T00:43:07.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.268+0000 7fd15ea00640 1 -- 172.21.15.123:0/2153766267 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164099770 con 0x55cc84f25050 2024-03-24T00:43:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2153766267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 msgr2=0x55cc84f2eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd16409b660 tx=0x7fd1640d1200 comp rx=0 tx=0).stop 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2153766267 shutdown_connections 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f2abb0 0x55cc84f23c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25a40 0x55cc84f36530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/2153766267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f25050 0x55cc84f2eff0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2153766267 >> 172.21.15.123:0/2153766267 conn(0x55cc84eed200 msgr2=0x55cc84f26cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2153766267 shutdown_connections 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- 172.21.15.123:0/2153766267 wait complete. 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.269+0000 7fd17bb7b740 1 -- start start 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 Processor -- start 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f34850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f35860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741a96e0 con 0x55cc84f25a40 2024-03-24T00:43:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741aaa00 con 0x55cc84f2abb0 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741a9d40 con 0x55cc84f25050 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd17bb7b740 10 client.0 _ll_register_callbacks cb 0x7fd16c044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd171e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd171e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50306/0 (socket says 172.21.15.123:50306) 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd171e00640 1 -- 172.21.15.123:0/3383783604 learned_addr learned my addr 172.21.15.123:0/3383783604 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd172800640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f35860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.270+0000 7fd171400640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f34850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 -- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 msgr2=0x55cc84f35860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f35860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 -- 172.21.15.123:0/3383783604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 msgr2=0x55cc84f34850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f34850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 -- 172.21.15.123:0/3383783604 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc84f35e50 con 0x55cc84f25050 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd172800640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f35860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd171e00640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1640cc880 tx=0x7fd1640cc7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37d78159eb7d2e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd153e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640b9060 con 0x55cc84f25050 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd1640999d0 con 0x55cc84f25050 2024-03-24T00:43:07.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.271+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd16409bdd0 con 0x55cc84f25050 2024-03-24T00:43:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1640b4040 con 0x55cc84f25050 2024-03-24T00:43:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 10 client.26056 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cc84f96db0 con 0x55cc84f25050 2024-03-24T00:43:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd16409bdd0 con 0x55cc84f25050 2024-03-24T00:43:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd1640a39e0 con 0x55cc84f25050 2024-03-24T00:43:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 10 client.26056 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 10 client.26056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 20 client.26056 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f974a0 0x55cc84f998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 4 client.26056 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:43:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cc84e91ae0 con 0x55cc84f974a0 2024-03-24T00:43:07.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.272+0000 7fd171400640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f974a0 0x55cc84f998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd171400640 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f974a0 0x55cc84f998c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a361991a2bcbbd server_cookie=e8c1bc85eb7cd19f in_seq=0 out_seq=0 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd153e00640 10 client.26056 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd153e00640 1 -- 172.21.15.123:0/3383783604 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cc84e91ae0 con 0x55cc84f974a0 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd153e00640 10 client.26056 handle_command_reply: tid=0 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd17bb7b740 1 client.26056 shutdown 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 msgr2=0x55cc84f34310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.273+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1640cc880 tx=0x7fd1640cc7d0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 shutdown_connections 2024-03-24T00:43:07.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc84f2abb0 0x55cc84f35860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc84f25a40 0x55cc84f34850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cc84f974a0 0x55cc84f998c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 --2- 172.21.15.123:0/3383783604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cc84f25050 0x55cc84f34310 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd153e00640 0 client.26056 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 >> 172.21.15.123:0/3383783604 conn(0x55cc84eed200 msgr2=0x55cc84f26440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 shutdown_connections 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 1 -- 172.21.15.123:0/3383783604 wait complete. 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.274+0000 7fd17bb7b740 20 client.26056 trim_cache size 0 max 16384 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:43:07.280 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:43:07.281 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:07.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 msgr2=0x7fd1580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 0x7fd1580a4d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd16c005e30 tx=0x7fd16c032590 comp rx=0 tx=0).stop 2024-03-24T00:43:07.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 msgr2=0x7fd174199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e9c0 tx=0x7fd16400ee80 comp rx=0 tx=0).stop 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 shutdown_connections 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd1580a2880 0x7fd1580a4d40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd174105370 0x7fd1741a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1741049b0 0x7fd174199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 --2- 172.21.15.123:0/1870834834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1741037c0 0x7fd174199a50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 >> 172.21.15.123:0/1870834834 conn(0x7fd1740fed50 msgr2=0x7fd174107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 shutdown_connections 2024-03-24T00:43:07.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.275+0000 7fd173200640 1 -- 172.21.15.123:0/1870834834 wait complete. 2024-03-24T00:43:07.293 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:43:07.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 -- 172.21.15.123:0/1066398463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec107b00 msgr2=0x7f42ec107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 --2- 172.21.15.123:0/1066398463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec107b00 0x7f42ec107f20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009860 tx=0x7f42e002bb70 comp rx=0 tx=0).stop 2024-03-24T00:43:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 -- 172.21.15.123:0/1066398463 shutdown_connections 2024-03-24T00:43:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 --2- 172.21.15.123:0/1066398463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec1096b0 0x7f42ec10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 --2- 172.21.15.123:0/1066398463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 0x7f42ec109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 --2- 172.21.15.123:0/1066398463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec107b00 0x7f42ec107f20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 -- 172.21.15.123:0/1066398463 >> 172.21.15.123:0/1066398463 conn(0x7f42ec076ab0 msgr2=0x7f42ec078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.493+0000 7f42f2600640 1 -- 172.21.15.123:0/1066398463 shutdown_connections 2024-03-24T00:43:07.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.494+0000 7f42f2600640 1 -- 172.21.15.123:0/1066398463 wait complete. 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.494+0000 7f42f2600640 1 Processor -- start 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 -- start start 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 0x7f42ec19e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec1096b0 0x7f42ec1a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec113780 con 0x7f42ec108cf0 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec113600 con 0x7f42ec107b00 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f2600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec113900 con 0x7f42ec1096b0 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38034/0 (socket says 172.21.15.123:38034) 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f1200640 1 -- 172.21.15.123:0/908289639 learned_addr learned my addr 172.21.15.123:0/908289639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42ebe00640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 0x7f42ec19e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.495+0000 7f42f1c00640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec1096b0 0x7f42ec1a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 -- 172.21.15.123:0/908289639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec1096b0 msgr2=0x7f42ec1a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec1096b0 0x7f42ec1a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 -- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 msgr2=0x7f42ec19e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 0x7f42ec19e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 -- 172.21.15.123:0/908289639 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec1a59d0 con 0x7f42ec107b00 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f1200640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f42e00349e0 tx=0x7f42e0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0041020 con 0x7f42ec107b00 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42e002c9f0 con 0x7f42ec107b00 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec1a5c60 con 0x7f42ec107b00 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.496+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0039040 con 0x7f42ec107b00 2024-03-24T00:43:07.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.497+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec1a61a0 con 0x7f42ec107b00 2024-03-24T00:43:07.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.498+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec10ee20 con 0x7f42ec107b00 2024-03-24T00:43:07.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.500+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f42e0044020 con 0x7f42ec107b00 2024-03-24T00:43:07.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.501+0000 7f42e9600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 0x7f42d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.501+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f42e00f0260 con 0x7f42ec107b00 2024-03-24T00:43:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.501+0000 7f42ebe00640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 0x7f42d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.502+0000 7f42ebe00640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 0x7f42d40a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42ec19f2c0 tx=0x7f42d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.505+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42e004b020 con 0x7f42ec107b00 2024-03-24T00:43:07.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:07 smithi123 ceph-mon[31487]: pgmap v542: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 45 MiB/s wr, 4.31k op/s 2024-03-24T00:43:07.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:07 smithi177 ceph-mon[36034]: pgmap v542: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 45 MiB/s wr, 4.31k op/s 2024-03-24T00:43:07.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.771+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42ec02d070 con 0x7f42ec107b00 2024-03-24T00:43:07.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.772+0000 7f42e9600640 1 -- 172.21.15.123:0/908289639 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f42e00bd2b0 con 0x7f42ec107b00 2024-03-24T00:43:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:43:07.772 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:43:07.773 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:43:07.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 msgr2=0x7f42d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 0x7f42d40a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42ec19f2c0 tx=0x7f42d8005f90 comp rx=0 tx=0).stop 2024-03-24T00:43:07.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 msgr2=0x7f42ec19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f42e00349e0 tx=0x7f42e0032430 comp rx=0 tx=0).stop 2024-03-24T00:43:07.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 shutdown_connections 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42ec1096b0 0x7f42ec1a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42d40a2880 0x7f42d40a4d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42ec108cf0 0x7f42ec19e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 --2- 172.21.15.123:0/908289639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107b00 0x7f42ec19de50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 >> 172.21.15.123:0/908289639 conn(0x7f42ec076ab0 msgr2=0x7f42ec10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 shutdown_connections 2024-03-24T00:43:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.775+0000 7f42f2600640 1 -- 172.21.15.123:0/908289639 wait complete. 2024-03-24T00:43:07.795 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:43:07.795 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:43:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:07 smithi204 ceph-mon[36345]: pgmap v542: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 45 MiB/s wr, 4.31k op/s 2024-03-24T00:43:07.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 -- 172.21.15.123:0/51478492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c1015a0 msgr2=0x7f515c1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 --2- 172.21.15.123:0/51478492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c1015a0 0x7f515c1019c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5150009900 tx=0x7f515002bc30 comp rx=0 tx=0).stop 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 -- 172.21.15.123:0/51478492 shutdown_connections 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 --2- 172.21.15.123:0/51478492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c103150 0x7f515c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 --2- 172.21.15.123:0/51478492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c102790 0x7f515c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 --2- 172.21.15.123:0/51478492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c1015a0 0x7f515c1019c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 -- 172.21.15.123:0/51478492 >> 172.21.15.123:0/51478492 conn(0x7f515c0fcb10 msgr2=0x7f515c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 -- 172.21.15.123:0/51478492 shutdown_connections 2024-03-24T00:43:07.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.997+0000 7f5162e00640 1 -- 172.21.15.123:0/51478492 wait complete. 2024-03-24T00:43:08.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 Processor -- start 2024-03-24T00:43:08.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 -- start start 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c102790 0x7f515c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c103150 0x7f515c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c10d260 con 0x7f515c102790 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c10d0e0 con 0x7f515c103150 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c10d3e0 con 0x7f515c1015a0 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5161a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5161a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50328/0 (socket says 172.21.15.123:50328) 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5161000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c102790 0x7f515c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5161a00640 1 -- 172.21.15.123:0/2051490988 learned_addr learned my addr 172.21.15.123:0/2051490988 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:07.999+0000 7f5162400640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c103150 0x7f515c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 -- 172.21.15.123:0/2051490988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c103150 msgr2=0x7f515c1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c103150 0x7f515c1a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 -- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c102790 msgr2=0x7f515c19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c102790 0x7f515c19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 -- 172.21.15.123:0/2051490988 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c1a16c0 con 0x7f515c1015a0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5161a00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5150000c00 tx=0x7f515002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f515a000640 1 -- 172.21.15.123:0/2051490988 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150043020 con 0x7f515c1015a0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f515a000640 1 -- 172.21.15.123:0/2051490988 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51500054e0 con 0x7f515c1015a0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c1a1950 con 0x7f515c1015a0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f515a000640 1 -- 172.21.15.123:0/2051490988 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150003c40 con 0x7f515c1015a0 2024-03-24T00:43:08.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.000+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c1a1ea0 con 0x7f515c1015a0 2024-03-24T00:43:08.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 Processor -- start 2024-03-24T00:43:08.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f515a000640 1 -- 172.21.15.123:0/2051490988 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5150046020 con 0x7f515c1015a0 2024-03-24T00:43:08.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 -- start start 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa687040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa68de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa5ea2c0 con 0x55b0fa687580 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa592c90 con 0x55b0fa6859f0 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.005+0000 7f516701e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa5e6360 con 0x55b0fa686bc0 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38064/0 (socket says 172.21.15.123:38064) 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 -- 172.21.15.123:0/3216718551 learned_addr learned my addr 172.21.15.123:0/3216718551 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5161000640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa68de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5161a00640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa687040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 -- 172.21.15.123:0/3216718551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 msgr2=0x55b0fa687040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa687040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 -- 172.21.15.123:0/3216718551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 msgr2=0x55b0fa68de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa68de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5162400640 1 -- 172.21.15.123:0/3216718551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0fa5b7080 con 0x55b0fa6859f0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.006+0000 7f5161000640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa68de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f5162400640 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5154009fd0 tx=0x7f51540118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4676f7313ee2d60a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f514ea00640 1 -- 172.21.15.123:0/3216718551 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154004d20 con 0x55b0fa6859f0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f514ea00640 1 -- 172.21.15.123:0/3216718551 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5154004050 con 0x55b0fa6859f0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f514ea00640 1 -- 172.21.15.123:0/3216718551 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154005260 con 0x55b0fa6859f0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f515a000640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f515a000640 1 -- 172.21.15.123:0/2051490988 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51500f0a50 con 0x7f515c1015a0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.007+0000 7f5161a00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- 172.21.15.123:0/3216718551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 msgr2=0x55b0fa685e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5154009fd0 tx=0x7f51540118d0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- 172.21.15.123:0/3216718551 shutdown_connections 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa68de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa687040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 --2- 172.21.15.123:0/3216718551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa6859f0 0x55b0fa685e10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- 172.21.15.123:0/3216718551 >> 172.21.15.123:0/3216718551 conn(0x55b0fa680de0 msgr2=0x55b0fa6811e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- 172.21.15.123:0/3216718551 shutdown_connections 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f5161a00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 0x7f51440a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f51500b9800 tx=0x7f51500b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- 172.21.15.123:0/3216718551 wait complete. 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.008+0000 7f516701e740 1 -- start start 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 Processor -- start 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa6859f0 0x55b0fa68fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa697240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa68e590 con 0x55b0fa6859f0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa5a60d0 con 0x55b0fa687580 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fa68e710 con 0x55b0fa686bc0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f516701e740 10 client.0 _ll_register_callbacks cb 0x55b0fa5aae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f5161000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.009+0000 7f5162400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa6859f0 0x55b0fa68fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50344/0 (socket says 172.21.15.123:50344) 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 -- 172.21.15.123:0/1300222951 learned_addr learned my addr 172.21.15.123:0/1300222951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5162400640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa697240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 -- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 msgr2=0x55b0fa697240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa697240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 -- 172.21.15.123:0/1300222951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa6859f0 msgr2=0x55b0fa68fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa6859f0 0x55b0fa68fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 -- 172.21.15.123:0/1300222951 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0fa2fe3f0 con 0x55b0fa686bc0 2024-03-24T00:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5162400640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa697240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5161000640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51480126e0 tx=0x7f5148012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a28c147a2afb0d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5143400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.010+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148004050 con 0x55b0fa686bc0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5154004050 con 0x55b0fa686bc0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51480041f0 con 0x55b0fa686bc0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51480043f0 con 0x55b0fa686bc0 2024-03-24T00:43:08.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 10 client.26074 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:08.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b0fa3c87d0 con 0x55b0fa686bc0 2024-03-24T00:43:08.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5154004050 con 0x55b0fa686bc0 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5148003c80 con 0x55b0fa686bc0 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 10 client.26074 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 10 client.26074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.011+0000 7f516701e740 20 client.26074 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa69b720 0x55b0fa69db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f516701e740 4 client.26074 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0fa5f7800 con 0x55b0fa69b720 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f5162400640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa69b720 0x55b0fa69db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f5162400640 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa69b720 0x55b0fa69db60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a1259ed4d539815 server_cookie=50de4b5b20d2b2da in_seq=0 out_seq=0 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.012+0000 7f5143400640 10 client.26074 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f5143400640 1 -- 172.21.15.123:0/1300222951 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b0fa5f7800 con 0x55b0fa69b720 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f5143400640 10 client.26074 handle_command_reply: tid=0 2024-03-24T00:43:08.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 client.26074 shutdown 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 msgr2=0x55b0fa690200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51480126e0 tx=0x7f5148012ba0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 shutdown_connections 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa697240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa69b720 0x55b0fa69db60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa686bc0 0x55b0fa690200 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 --2- 172.21.15.123:0/1300222951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa6859f0 0x55b0fa68fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f5143400640 0 client.26074 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 >> 172.21.15.123:0/1300222951 conn(0x55b0fa680de0 msgr2=0x55b0fa680160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.013+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 shutdown_connections 2024-03-24T00:43:08.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.014+0000 7f516701e740 1 -- 172.21.15.123:0/1300222951 wait complete. 2024-03-24T00:43:08.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.014+0000 7f516701e740 20 client.26074 trim_cache size 0 max 16384 2024-03-24T00:43:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 Processor -- start 2024-03-24T00:43:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 -- start start 2024-03-24T00:43:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa686bc0 0x55b0fa697200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa68a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a8af0 con 0x55b0fa686bc0 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1ab0b0 con 0x55b0fa687580 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f516701e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a90f0 con 0x55b0fa6859f0 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f5162400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f5162400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50358/0 (socket says 172.21.15.123:50358) 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f5162400640 1 -- 172.21.15.123:0/2705139373 learned_addr learned my addr 172.21.15.123:0/2705139373 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f5162400640 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa68a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.028+0000 7f5161000640 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa686bc0 0x55b0fa697200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 -- 172.21.15.123:0/2705139373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 msgr2=0x55b0fa68a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa68a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 -- 172.21.15.123:0/2705139373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa686bc0 msgr2=0x55b0fa697200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa686bc0 0x55b0fa697200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 -- 172.21.15.123:0/2705139373 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0fa690cb0 con 0x55b0fa6859f0 2024-03-24T00:43:08.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f5162400640 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f51540095e0 tx=0x7f5154005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b7e79272a8fa05e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f514e000640 1 -- 172.21.15.123:0/2705139373 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51540041c0 con 0x55b0fa6859f0 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f514e000640 1 -- 172.21.15.123:0/2705139373 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5154004840 con 0x55b0fa6859f0 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.029+0000 7f514e000640 1 -- 172.21.15.123:0/2705139373 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515400fd50 con 0x55b0fa6859f0 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.030+0000 7f516701e740 1 -- 172.21.15.123:0/2705139373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 msgr2=0x55b0fa689df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.030+0000 7f516701e740 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f51540095e0 tx=0x7f5154005fd0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 -- 172.21.15.123:0/2705139373 shutdown_connections 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa687580 0x55b0fa68a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa686bc0 0x55b0fa697200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 --2- 172.21.15.123:0/2705139373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa689df0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 -- 172.21.15.123:0/2705139373 >> 172.21.15.123:0/2705139373 conn(0x55b0fa680de0 msgr2=0x55b0fa6848d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 -- 172.21.15.123:0/2705139373 shutdown_connections 2024-03-24T00:43:08.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 -- 172.21.15.123:0/2705139373 wait complete. 2024-03-24T00:43:08.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 -- start start 2024-03-24T00:43:08.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.031+0000 7f516701e740 1 Processor -- start 2024-03-24T00:43:08.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa694fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 0x55b0fa6954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa696500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a9730 con 0x55b0fa687580 2024-03-24T00:43:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1aaa50 con 0x55b0fa686bc0 2024-03-24T00:43:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a9d90 con 0x55b0fa6859f0 2024-03-24T00:43:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f516701e740 10 client.0 _ll_register_callbacks cb 0x7f5148009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5162400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa694fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 0x55b0fa6954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5162400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa694fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50374/0 (socket says 172.21.15.123:50374) 2024-03-24T00:43:08.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5162400640 1 -- 172.21.15.123:0/2590888828 learned_addr learned my addr 172.21.15.123:0/2590888828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:08.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5162400640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa696500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 -- 172.21.15.123:0/2590888828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 msgr2=0x55b0fa694fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa694fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 -- 172.21.15.123:0/2590888828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 msgr2=0x55b0fa696500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa696500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5161000640 1 -- 172.21.15.123:0/2590888828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0fa696af0 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.032+0000 7f5162400640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa696500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f5161000640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 0x55b0fa6954f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5148010430 tx=0x7f5148010d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38716eb15c6f9101 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f5143400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148035060 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5148005030 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5154004840 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f514802c040 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f516701e740 10 client.26075 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.033+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b0fa6f7700 con 0x55b0fa686bc0 2024-03-24T00:43:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5154004840 con 0x55b0fa686bc0 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5148004590 con 0x55b0fa686bc0 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 10 client.26075 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 10 client.26075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 20 client.26075 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa6f8020 0x55b0fa6fa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 4 client.26075 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b0fa5f1b50 con 0x55b0fa6f8020 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f5162400640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa6f8020 0x55b0fa6fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f5143400640 10 client.26075 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.034+0000 7f5162400640 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa6f8020 0x55b0fa6fa440 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd89e9395c38a6bb server_cookie=63a6215bd1192568 in_seq=0 out_seq=0 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f5143400640 1 -- 172.21.15.123:0/2590888828 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b0fa5f1b50 con 0x55b0fa6f8020 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f5143400640 10 client.26075 handle_command_reply: tid=0 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 client.26075 shutdown 2024-03-24T00:43:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 msgr2=0x55b0fa6954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 0x55b0fa6954f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5148010430 tx=0x7f5148010d00 comp rx=0 tx=0).stop 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 shutdown_connections 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b0fa687580 0x55b0fa696500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0fa686bc0 0x55b0fa6954f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55b0fa6f8020 0x55b0fa6fa440 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 --2- 172.21.15.123:0/2590888828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b0fa6859f0 0x55b0fa694fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 >> 172.21.15.123:0/2590888828 conn(0x55b0fa680de0 msgr2=0x55b0fa67f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f5143400640 0 client.26075 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 shutdown_connections 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 1 -- 172.21.15.123:0/2590888828 wait complete. 2024-03-24T00:43:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.035+0000 7f516701e740 20 client.26075 trim_cache size 0 max 16384 2024-03-24T00:43:08.045 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:43:08.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 msgr2=0x7f51440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 0x7f51440a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f51500b9800 tx=0x7f51500b9040 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 msgr2=0x7f515c199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5150000c00 tx=0x7f515002c650 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 shutdown_connections 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c103150 0x7f515c1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f515c102790 0x7f515c19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 --2- 172.21.15.123:0/2051490988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f515c1015a0 0x7f515c199b40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 >> 172.21.15.123:0/2051490988 conn(0x7f515c0fcb10 msgr2=0x7f515c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 shutdown_connections 2024-03-24T00:43:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:08.036+0000 7f5162e00640 1 -- 172.21.15.123:0/2051490988 wait complete. 2024-03-24T00:43:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:08 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/908289639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:08 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/908289639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:08.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:08 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/908289639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:09.055 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:43:09.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:09.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:43:09.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 -- 172.21.15.123:0/2576155563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a0102170 msgr2=0x7fe3a0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/2576155563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a0102170 0x7fe3a0108a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe398009860 tx=0x7fe39802bb70 comp rx=0 tx=0).stop 2024-03-24T00:43:09.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 -- 172.21.15.123:0/2576155563 shutdown_connections 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/2576155563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a0102170 0x7fe3a0108a00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/2576155563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01017b0 0x7fe3a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/2576155563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a01005c0 0x7fe3a01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 -- 172.21.15.123:0/2576155563 >> 172.21.15.123:0/2576155563 conn(0x7fe3a00fbb50 msgr2=0x7fe3a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.250+0000 7fe3a4e00640 1 -- 172.21.15.123:0/2576155563 shutdown_connections 2024-03-24T00:43:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.251+0000 7fe3a4e00640 1 -- 172.21.15.123:0/2576155563 wait complete. 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.251+0000 7fe3a4e00640 1 Processor -- start 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 -- start start 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 0x7fe3a01a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a0102170 0x7fe3a01a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a010c220 con 0x7fe3a0102170 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a010c0a0 con 0x7fe3a01005c0 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe3a4e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a010c3a0 con 0x7fe3a01017b0 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe395400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 0x7fe3a01a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 0x7fe3a01a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38108/0 (socket says 172.21.15.123:38108) 2024-03-24T00:43:09.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe39de00640 1 -- 172.21.15.123:0/4018075884 learned_addr learned my addr 172.21.15.123:0/4018075884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe395400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50396/0 (socket says 172.21.15.123:50396) 2024-03-24T00:43:09.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.252+0000 7fe39e800640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a0102170 0x7fe3a01a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 -- 172.21.15.123:0/4018075884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 msgr2=0x7fe3a01a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 0x7fe3a01a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 -- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a0102170 msgr2=0x7fe3a01a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a0102170 0x7fe3a01a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 -- 172.21.15.123:0/4018075884 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a01a5be0 con 0x7fe3a01017b0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe395400640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe38800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe397400640 1 -- 172.21.15.123:0/4018075884 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38800cd00 con 0x7fe3a01017b0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe397400640 1 -- 172.21.15.123:0/4018075884 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3880035f0 con 0x7fe3a01017b0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a01a5ed0 con 0x7fe3a01017b0 2024-03-24T00:43:09.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe397400640 1 -- 172.21.15.123:0/4018075884 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388010670 con 0x7fe3a01017b0 2024-03-24T00:43:09.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.253+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a01a63e0 con 0x7fe3a01017b0 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 Processor -- start 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 -- start start 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725b994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 0x55f725b9e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 0x55f725ba1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725afeeb0 con 0x55f725b9ec00 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725aa7a60 con 0x55f725b990a0 2024-03-24T00:43:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725afaf50 con 0x55f725b99a90 2024-03-24T00:43:09.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe397400640 1 -- 172.21.15.123:0/4018075884 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe3880037a0 con 0x7fe3a01017b0 2024-03-24T00:43:09.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe39e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725b994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.257+0000 7fe39e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725b994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38124/0 (socket says 172.21.15.123:38124) 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39e800640 1 -- 172.21.15.123:0/3906312034 learned_addr learned my addr 172.21.15.123:0/3906312034 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 0x55f725b9e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe395400640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 0x55f725ba1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 -- 172.21.15.123:0/3906312034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 msgr2=0x55f725b994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725b994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 -- 172.21.15.123:0/3906312034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 msgr2=0x55f725ba1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 0x55f725ba1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 -- 172.21.15.123:0/3906312034 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7258db7e0 con 0x55f725b99a90 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe395400640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 0x55f725ba1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.258+0000 7fe39de00640 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 0x55f725b9e590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe390009fd0 tx=0x7fe3900118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed62285fd90c9162 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.259+0000 7fe38f400640 1 -- 172.21.15.123:0/3906312034 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390004b20 con 0x55f725b99a90 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.259+0000 7fe38f400640 1 -- 172.21.15.123:0/3906312034 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe390003e50 con 0x55f725b99a90 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.259+0000 7fe38f400640 1 -- 172.21.15.123:0/3906312034 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000f5a0 con 0x55f725b99a90 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.259+0000 7fe397400640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.259+0000 7fe397400640 1 -- 172.21.15.123:0/4018075884 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe3880cebf0 con 0x7fe3a01017b0 2024-03-24T00:43:09.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe39e800640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3906312034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 msgr2=0x55f725b9e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 0x55f725b9e590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe390009fd0 tx=0x7fe3900118d0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3906312034 shutdown_connections 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b9ec00 0x55f725ba1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b99a90 0x55f725b9e590 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3906312034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725b994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3906312034 >> 172.21.15.123:0/3906312034 conn(0x55f725b94610 msgr2=0x55f725b94a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3906312034 shutdown_connections 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3906312034 wait complete. 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.260+0000 7fe39e800640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 0x7fe3840a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe3980057c0 tx=0x7fe3980056d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.261+0000 7fe3a7b71740 1 -- start start 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.261+0000 7fe3a7b71740 1 Processor -- start 2024-03-24T00:43:09.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725ba31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725baa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725ba1bb0 con 0x55f725b99a90 2024-03-24T00:43:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725abaeb0 con 0x55f725b990a0 2024-03-24T00:43:09.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f725ba1d30 con 0x55f725b9ec00 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe3a7b71740 10 client.0 _ll_register_callbacks cb 0x55f725aaf150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725ba31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe39e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725baa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725ba31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38128/0 (socket says 172.21.15.123:38128) 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe39de00640 1 -- 172.21.15.123:0/3293584820 learned_addr learned my addr 172.21.15.123:0/3293584820 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.262+0000 7fe395400640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 -- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 msgr2=0x55f725ba31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725ba31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 -- 172.21.15.123:0/3293584820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 msgr2=0x55f725ba3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 -- 172.21.15.123:0/3293584820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f72593bf70 con 0x55f725b9ec00 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe39e800640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725baa770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe398009830 tx=0x7fe39802bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54605b6432a035b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.263+0000 7fe383e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39804e020 con 0x55f725b9ec00 2024-03-24T00:43:09.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe39802c710 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe390003e50 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe398039040 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 10 client.26092 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f7257d3170 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe390003e50 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3980035f0 con 0x55f725b9ec00 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 10 client.26092 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 10 client.26092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.264+0000 7fe3a7b71740 20 client.26092 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725baeff0 0x55f725bb13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe3a7b71740 4 client.26092 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f725b0c3f0 con 0x55f725baeff0 2024-03-24T00:43:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe39de00640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725baeff0 0x55f725bb13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe39de00640 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725baeff0 0x55f725bb13e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb712d7438fce0b3 server_cookie=3bbb07c7790c16bc in_seq=0 out_seq=0 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.265+0000 7fe383e00640 10 client.26092 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.267+0000 7fe383e00640 1 -- 172.21.15.123:0/3293584820 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f725b0c3f0 con 0x55f725baeff0 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.267+0000 7fe383e00640 10 client.26092 handle_command_reply: tid=0 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.267+0000 7fe3a7b71740 1 client.26092 shutdown 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 msgr2=0x55f725baa770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725baa770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe398009830 tx=0x7fe39802bb70 comp rx=0 tx=0).stop 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 shutdown_connections 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725baeff0 0x55f725bb13e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725baa770 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3293584820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725ba31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe383e00640 0 client.26092 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 >> 172.21.15.123:0/3293584820 conn(0x55f725b94610 msgr2=0x55f725b93830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 shutdown_connections 2024-03-24T00:43:09.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3293584820 wait complete. 2024-03-24T00:43:09.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.268+0000 7fe3a7b71740 20 client.26092 trim_cache size 0 max 16384 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.287+0000 7fe3a7b71740 1 Processor -- start 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.287+0000 7fe3a7b71740 1 -- start start 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.287+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.287+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725bb5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01ad190 con 0x55f725b990a0 2024-03-24T00:43:09.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01af750 con 0x55f725b99a90 2024-03-24T00:43:09.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01ad790 con 0x55f725b9ec00 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe395400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe395400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38144/0 (socket says 172.21.15.123:38144) 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:32890/0 (socket says 172.21.15.123:32890) 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe395400640 1 -- 172.21.15.123:0/2298557466 learned_addr learned my addr 172.21.15.123:0/2298557466 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725bb5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 -- 172.21.15.123:0/2298557466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 msgr2=0x55f725bb5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725bb5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 -- 172.21.15.123:0/2298557466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 msgr2=0x55f725baa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39de00640 1 -- 172.21.15.123:0/2298557466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f725ba2fe0 con 0x55f725b990a0 2024-03-24T00:43:09.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe395400640 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:09.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.288+0000 7fe39e800640 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725bb5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:09.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.289+0000 7fe39de00640 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe390000f80 tx=0x7fe390009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23d71f1f7159091 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.289+0000 7fe38ea00640 1 -- 172.21.15.123:0/2298557466 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39001d040 con 0x55f725b990a0 2024-03-24T00:43:09.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.289+0000 7fe38ea00640 1 -- 172.21.15.123:0/2298557466 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39002a070 con 0x55f725b990a0 2024-03-24T00:43:09.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.289+0000 7fe38ea00640 1 -- 172.21.15.123:0/2298557466 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000f5a0 con 0x55f725b990a0 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- 172.21.15.123:0/2298557466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 msgr2=0x55f725b97ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe390000f80 tx=0x7fe390009290 comp rx=0 tx=0).stop 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- 172.21.15.123:0/2298557466 shutdown_connections 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725bb5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b99a90 0x55f725baa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/2298557466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b990a0 0x55f725b97ee0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- 172.21.15.123:0/2298557466 >> 172.21.15.123:0/2298557466 conn(0x55f725b94610 msgr2=0x55f725b93830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- 172.21.15.123:0/2298557466 shutdown_connections 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- 172.21.15.123:0/2298557466 wait complete. 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.290+0000 7fe3a7b71740 1 -- start start 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 Processor -- start 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725ba8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01addd0 con 0x55f725b99a90 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01af0f0 con 0x55f725b990a0 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a01ae430 con 0x55f725b9ec00 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe3a7b71740 10 client.0 _ll_register_callbacks cb 0x7fe398034440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe39de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38154/0 (socket says 172.21.15.123:38154) 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe39de00640 1 -- 172.21.15.123:0/3036355408 learned_addr learned my addr 172.21.15.123:0/3036355408 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.291+0000 7fe395400640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39e800640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725ba8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 -- 172.21.15.123:0/3036355408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 msgr2=0x55f725ba8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725ba8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 -- 172.21.15.123:0/3036355408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 msgr2=0x55f725ba8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 -- 172.21.15.123:0/3036355408 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f725ba95c0 con 0x55f725b990a0 2024-03-24T00:43:09.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe395400640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe39de00640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe390004730 tx=0x7fe390009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd10479d3b67bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.292+0000 7fe383e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390031040 con 0x55f725b990a0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe39002a070 con 0x55f725b990a0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe390004320 con 0x55f725b990a0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe390003ab0 con 0x55f725b990a0 2024-03-24T00:43:09.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe3a7b71740 10 client.26093 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f725c0b700 con 0x55f725b990a0 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.293+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39002a070 con 0x55f725b990a0 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe390034070 con 0x55f725b990a0 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 10 client.26093 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 20 client.26093 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725c0bf70 0x55f725c0e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 4 client.26093 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f725b06740 con 0x55f725c0bf70 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.294+0000 7fe395400640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725c0bf70 0x55f725c0e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.295+0000 7fe383e00640 10 client.26093 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.295+0000 7fe395400640 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725c0bf70 0x55f725c0e390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e93cd7b08df96be6 server_cookie=f839cad15381a5d0 in_seq=0 out_seq=0 2024-03-24T00:43:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.442+0000 7fe383e00640 1 -- 172.21.15.123:0/3036355408 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f725b06740 con 0x55f725c0bf70 2024-03-24T00:43:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.442+0000 7fe383e00640 10 client.26093 handle_command_reply: tid=0 2024-03-24T00:43:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.442+0000 7fe3a7b71740 1 client.26093 shutdown 2024-03-24T00:43:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 msgr2=0x55f725baa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe390004730 tx=0x7fe390009f00 comp rx=0 tx=0).stop 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 shutdown_connections 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f725c0bf70 0x55f725c0e390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f725b9ec00 0x55f725ba8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe383e00640 0 client.26093 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f725b99a90 0x55f725ba8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 --2- 172.21.15.123:0/3036355408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f725b990a0 0x55f725baa430 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.443+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 >> 172.21.15.123:0/3036355408 conn(0x55f725b94610 msgr2=0x55f725b9a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.444+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 shutdown_connections 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.444+0000 7fe3a7b71740 1 -- 172.21.15.123:0/3036355408 wait complete. 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.444+0000 7fe3a7b71740 20 client.26093 trim_cache size 0 max 16384 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "642215cf-d331-46ca-9914-6bce39b88e00", 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:43:09.444 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 msgr2=0x7fe3840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 0x7fe3840a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe3980057c0 tx=0x7fe3980056d0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 msgr2=0x7fe3a01a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe38800ee50 comp rx=0 tx=0).stop 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 shutdown_connections 2024-03-24T00:43:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3a0102170 0x7fe3a01a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3a01017b0 0x7fe3a01a2c60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 --2- 172.21.15.123:0/4018075884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a01005c0 0x7fe3a01a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.445+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 >> 172.21.15.123:0/4018075884 conn(0x7fe3a00fbb50 msgr2=0x7fe3a00fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.446+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 shutdown_connections 2024-03-24T00:43:09.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.446+0000 7fe3a4e00640 1 -- 172.21.15.123:0/4018075884 wait complete. 2024-03-24T00:43:09.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:43:09.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:43:09.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 -- 172.21.15.123:0/341578634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c102000 msgr2=0x7f869c1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 --2- 172.21.15.123:0/341578634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c102000 0x7f869c1043f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f868c009900 tx=0x7f868c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 -- 172.21.15.123:0/341578634 shutdown_connections 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 --2- 172.21.15.123:0/341578634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c107260 0x7f869c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 --2- 172.21.15.123:0/341578634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 0x7f869c106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 --2- 172.21.15.123:0/341578634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c102000 0x7f869c1043f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 -- 172.21.15.123:0/341578634 >> 172.21.15.123:0/341578634 conn(0x7f869c0fba10 msgr2=0x7f869c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 -- 172.21.15.123:0/341578634 shutdown_connections 2024-03-24T00:43:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.673+0000 7f869be00640 1 -- 172.21.15.123:0/341578634 wait complete. 2024-03-24T00:43:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.674+0000 7f869be00640 1 Processor -- start 2024-03-24T00:43:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 -- start start 2024-03-24T00:43:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 0x7f869c0fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107260 0x7f869c1004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c10d260 con 0x7f869c104930 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c10d0e0 con 0x7f869c107260 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c10d3e0 con 0x7f869c102000 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50442/0 (socket says 172.21.15.123:50442) 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 0x7f869c0fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107260 0x7f869c1004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.675+0000 7f869aa00640 1 -- 172.21.15.123:0/1581538252 learned_addr learned my addr 172.21.15.123:0/1581538252 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 -- 172.21.15.123:0/1581538252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107260 msgr2=0x7f869c1004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107260 0x7f869c1004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 -- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 msgr2=0x7f869c0fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 0x7f869c0fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 -- 172.21.15.123:0/1581538252 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c100d80 con 0x7f869c102000 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869aa00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f868c000c00 tx=0x7f868c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f8693400640 1 -- 172.21.15.123:0/1581538252 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c043020 con 0x7f869c102000 2024-03-24T00:43:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f8693400640 1 -- 172.21.15.123:0/1581538252 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c0054e0 con 0x7f869c102000 2024-03-24T00:43:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.676+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c1a19a0 con 0x7f869c102000 2024-03-24T00:43:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.677+0000 7f8693400640 1 -- 172.21.15.123:0/1581538252 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c003c40 con 0x7f869c102000 2024-03-24T00:43:09.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.677+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c1a1f40 con 0x7f869c102000 2024-03-24T00:43:09.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f86a43f2740 1 Processor -- start 2024-03-24T00:43:09.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f8693400640 1 -- 172.21.15.123:0/1581538252 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f868c046020 con 0x7f869c102000 2024-03-24T00:43:09.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f86a43f2740 1 -- start start 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a4050 0x56347f7a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7a9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.681+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 0x56347f7ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f86a43f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f709250 con 0x56347f7a4050 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f86a43f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f6b1a30 con 0x56347f7a9bb0 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f86a43f2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f7052f0 con 0x56347f7a4a40 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 0x56347f7ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a4050 0x56347f7a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 0x56347f7ac040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38184/0 (socket says 172.21.15.123:38184) 2024-03-24T00:43:09.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869a000640 1 -- 172.21.15.123:0/3814299035 learned_addr learned my addr 172.21.15.123:0/3814299035 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 -- 172.21.15.123:0/3814299035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 msgr2=0x56347f7ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 0x56347f7ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 -- 172.21.15.123:0/3814299035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a4050 msgr2=0x56347f7a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a4050 0x56347f7a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.682+0000 7f869aa00640 1 -- 172.21.15.123:0/3814299035 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347f69c690 con 0x56347f7a4a40 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f869aa00640 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7a9540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f868c0ee4c0 tx=0x7f868c0f6b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be2912f1fc942033 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f8693400640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 0x7f86800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f8687e00640 1 -- 172.21.15.123:0/3814299035 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0ff070 con 0x56347f7a4a40 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f8687e00640 1 -- 172.21.15.123:0/3814299035 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c0bdac0 con 0x56347f7a4a40 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f8687e00640 1 -- 172.21.15.123:0/3814299035 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0f0e60 con 0x56347f7a4a40 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f8693400640 1 -- 172.21.15.123:0/1581538252 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f868c0f0a70 con 0x7f869c102000 2024-03-24T00:43:09.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.683+0000 7f869b400640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 0x7f86800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 -- 172.21.15.123:0/3814299035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 msgr2=0x56347f7a9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f869b400640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 0x7f86800a4d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56347f7a52a0 tx=0x7f8694009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7a9540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f868c0ee4c0 tx=0x7f868c0f6b00 comp rx=0 tx=0).stop 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 -- 172.21.15.123:0/3814299035 shutdown_connections 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a9bb0 0x56347f7ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7a9540 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 --2- 172.21.15.123:0/3814299035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a4050 0x56347f7a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 -- 172.21.15.123:0/3814299035 >> 172.21.15.123:0/3814299035 conn(0x56347f76c200 msgr2=0x56347f79f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 -- 172.21.15.123:0/3814299035 shutdown_connections 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.684+0000 7f86a43f2740 1 -- 172.21.15.123:0/3814299035 wait complete. 2024-03-24T00:43:09.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.685+0000 7f86a43f2740 1 -- start start 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.685+0000 7f86a43f2740 1 Processor -- start 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f7ac920 con 0x56347f7a9bb0 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f6c4e80 con 0x56347f7a4a40 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f7acaa0 con 0x56347f7a4050 2024-03-24T00:43:09.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f86a43f2740 10 client.0 _ll_register_callbacks cb 0x56347f6d7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38196/0 (socket says 172.21.15.123:38196) 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869a000640 1 -- 172.21.15.123:0/525138658 learned_addr learned my addr 172.21.15.123:0/525138658 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50472/0 (socket says 172.21.15.123:50472) 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.686+0000 7f869b400640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 -- 172.21.15.123:0/525138658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 msgr2=0x56347f7add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 -- 172.21.15.123:0/525138658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 msgr2=0x56347f7b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 -- 172.21.15.123:0/525138658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56347f4e77d0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869aa00640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f869a000640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8688012740 tx=0x7f8688012c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d10cd89e022fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f8684c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.687+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688004030 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f868c0bdac0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f86880041d0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86880043d0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 10 client.26105 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56347f41d3f0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c0bdac0 con 0x56347f7a4a40 2024-03-24T00:43:09.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8688003c70 con 0x56347f7a4a40 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 10 client.26105 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 10 client.26105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 20 client.26105 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f7b9ab0 0x56347f7bbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 4 client.26105 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.688+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56347f716790 con 0x56347f7b9ab0 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.689+0000 7f869aa00640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f7b9ab0 0x56347f7bbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.689+0000 7f8684c00640 10 client.26105 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.689+0000 7f869aa00640 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f7b9ab0 0x56347f7bbf00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da37749974b6763 server_cookie=d885ee2d1f9b903e in_seq=0 out_seq=0 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.690+0000 7f8684c00640 1 -- 172.21.15.123:0/525138658 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56347f716790 con 0x56347f7b9ab0 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.690+0000 7f8684c00640 10 client.26105 handle_command_reply: tid=0 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.690+0000 7f86a43f2740 1 client.26105 shutdown 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 msgr2=0x56347f7ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8688012740 tx=0x7f8688012c00 comp rx=0 tx=0).stop 2024-03-24T00:43:09.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 shutdown_connections 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4a40 0x56347f7ae260 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f7b9ab0 0x56347f7bbf00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 --2- 172.21.15.123:0/525138658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4050 0x56347f7add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f8684c00640 0 client.26105 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 >> 172.21.15.123:0/525138658 conn(0x56347f76c200 msgr2=0x56347f79ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 shutdown_connections 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 1 -- 172.21.15.123:0/525138658 wait complete. 2024-03-24T00:43:09.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.691+0000 7f86a43f2740 20 client.26105 trim_cache size 0 max 16384 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 Processor -- start 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 -- start start 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f79e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a4640 con 0x56347f7a9bb0 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a6c00 con 0x56347f7a4050 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.710+0000 7f86a43f2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a4c40 con 0x56347f7a4a40 2024-03-24T00:43:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50484/0 (socket says 172.21.15.123:50484) 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38200/0 (socket says 172.21.15.123:38200) 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 -- 172.21.15.123:0/781569628 learned_addr learned my addr 172.21.15.123:0/781569628 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869b400640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f79e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 -- 172.21.15.123:0/781569628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 msgr2=0x56347f7a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 -- 172.21.15.123:0/781569628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 msgr2=0x56347f79e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f79e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 -- 172.21.15.123:0/781569628 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347f3dd0f0 con 0x56347f7a4a40 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869aa00640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869b400640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f79e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:09 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:09.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:09 smithi177 ceph-mon[36034]: pgmap v543: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 48 MiB/s wr, 4.63k op/s 2024-03-24T00:43:09.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.711+0000 7f869a000640 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688012320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7364ec5c005e6f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.712+0000 7f8687400640 1 -- 172.21.15.123:0/781569628 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688003fe0 con 0x56347f7a4a40 2024-03-24T00:43:09.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.712+0000 7f8687400640 1 -- 172.21.15.123:0/781569628 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8688012e50 con 0x56347f7a4a40 2024-03-24T00:43:09.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.712+0000 7f8687400640 1 -- 172.21.15.123:0/781569628 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688010c40 con 0x56347f7a4a40 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- 172.21.15.123:0/781569628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 msgr2=0x56347f7b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688012320 comp rx=0 tx=0).stop 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- 172.21.15.123:0/781569628 shutdown_connections 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f79e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b51f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 --2- 172.21.15.123:0/781569628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- 172.21.15.123:0/781569628 >> 172.21.15.123:0/781569628 conn(0x56347f76c200 msgr2=0x56347f7a2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- 172.21.15.123:0/781569628 shutdown_connections 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- 172.21.15.123:0/781569628 wait complete. 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 -- start start 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.713+0000 7f86a43f2740 1 Processor -- start 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a5280 con 0x56347f7a9bb0 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a65a0 con 0x56347f7a4050 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1a58e0 con 0x56347f7a4a40 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f86a43f2740 10 client.0 _ll_register_callbacks cb 0x7f8688009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:38216/0 (socket says 172.21.15.123:38216) 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 -- 172.21.15.123:0/47351035 learned_addr learned my addr 172.21.15.123:0/47351035 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869b400640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869a000640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 -- 172.21.15.123:0/47351035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 msgr2=0x56347f7b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 -- 172.21.15.123:0/47351035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 msgr2=0x56347f7b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.714+0000 7f869aa00640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f869aa00640 1 -- 172.21.15.123:0/47351035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56347f7b49f0 con 0x56347f7a4050 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f869a000640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:09.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f869aa00640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b9d40 tx=0x7f868c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f6598cd50b6d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f8684c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0ff040 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8688012e50 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f868c004430 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f868c0f0e60 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f86a43f2740 10 client.26111 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56347f815980 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.715+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8688012e50 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f868c0fa040 con 0x56347f7a4050 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 10 client.26111 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 10 client.26111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 20 client.26111 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f816090 0x56347f8184b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 4 client.26111 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56347f710ae0 con 0x56347f816090 2024-03-24T00:43:09.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.716+0000 7f869a000640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f816090 0x56347f8184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f8684c00640 10 client.26111 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f869a000640 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f816090 0x56347f8184b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828fdae4da9a8504 server_cookie=5f457d501937d318 in_seq=0 out_seq=0 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f8684c00640 1 -- 172.21.15.123:0/47351035 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x56347f710ae0 con 0x56347f816090 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f8684c00640 10 client.26111 handle_command_reply: tid=0 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 client.26111 shutdown 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 msgr2=0x56347f7b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b9d40 tx=0x7f868c009260 comp rx=0 tx=0).stop 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 shutdown_connections 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56347f7a9bb0 0x56347f7b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56347f816090 0x56347f8184b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56347f7a4a40 0x56347f7b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 --2- 172.21.15.123:0/47351035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347f7a4050 0x56347f7b2eb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 >> 172.21.15.123:0/47351035 conn(0x56347f76c200 msgr2=0x56347f7a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f8684c00640 0 client.26111 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:09.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 shutdown_connections 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 1 -- 172.21.15.123:0/47351035 wait complete. 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.717+0000 7f86a43f2740 20 client.26111 trim_cache size 0 max 16384 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "642215cf-d331-46ca-9914-6bce39b88e00": { 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "642215cf-d331-46ca-9914-6bce39b88e00", 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:43:09.722 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:43:09.723 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:09.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 msgr2=0x7f86800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 0x7f86800a4d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56347f7a52a0 tx=0x7f8694009500 comp rx=0 tx=0).stop 2024-03-24T00:43:09.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 msgr2=0x7f869c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f868c000c00 tx=0x7f868c02c650 comp rx=0 tx=0).stop 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 shutdown_connections 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869c107260 0x7f869c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f86800a2880 0x7f86800a4d40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f869c104930 0x7f869c0fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 --2- 172.21.15.123:0/1581538252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f869c102000 0x7f869c101970 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 >> 172.21.15.123:0/1581538252 conn(0x7f869c0fba10 msgr2=0x7f869c1028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 shutdown_connections 2024-03-24T00:43:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:09.718+0000 7f869be00640 1 -- 172.21.15.123:0/1581538252 wait complete. 2024-03-24T00:43:09.732 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:09 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:09.732 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:09 smithi123 ceph-mon[31487]: pgmap v543: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 48 MiB/s wr, 4.63k op/s 2024-03-24T00:43:09.734 INFO:tasks.cephfs.filesystem:scrub status for tag:642215cf-d331-46ca-9914-6bce39b88e00 - {'path': '/', 'tag': '642215cf-d331-46ca-9914-6bce39b88e00', 'options': 'recursive,force'} 2024-03-24T00:43:09.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:09 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:09.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:09 smithi204 ceph-mon[36345]: pgmap v543: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 48 MiB/s wr, 4.63k op/s 2024-03-24T00:43:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:43:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:43:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:10.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:10 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:10 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:10 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:43:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:11 smithi204 ceph-mon[36345]: pgmap v544: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 51 MiB/s wr, 4.99k op/s 2024-03-24T00:43:11.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:11 smithi123 ceph-mon[31487]: pgmap v544: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 51 MiB/s wr, 4.99k op/s 2024-03-24T00:43:11.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:11 smithi177 ceph-mon[36034]: pgmap v544: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 51 MiB/s wr, 4.99k op/s 2024-03-24T00:43:14.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:14 smithi123 ceph-mon[31487]: pgmap v545: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 44 MiB/s wr, 4.30k op/s 2024-03-24T00:43:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:14 smithi177 ceph-mon[36034]: pgmap v545: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 44 MiB/s wr, 4.30k op/s 2024-03-24T00:43:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:14 smithi204 ceph-mon[36345]: pgmap v545: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 44 MiB/s wr, 4.30k op/s 2024-03-24T00:43:16.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:16 smithi123 ceph-mon[31487]: pgmap v546: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 39 MiB/s wr, 3.81k op/s 2024-03-24T00:43:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:16 smithi177 ceph-mon[36034]: pgmap v546: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 39 MiB/s wr, 3.81k op/s 2024-03-24T00:43:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:16 smithi204 ceph-mon[36345]: pgmap v546: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 39 MiB/s wr, 3.81k op/s 2024-03-24T00:43:17.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:17 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:17.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:17 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:43:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:17 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:17.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:17 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:43:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:17 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:17 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:43:18.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:18 smithi123 ceph-mon[31487]: pgmap v547: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 3.12k op/s 2024-03-24T00:43:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:18 smithi177 ceph-mon[36034]: pgmap v547: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 3.12k op/s 2024-03-24T00:43:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:18 smithi204 ceph-mon[36345]: pgmap v547: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 32 MiB/s wr, 3.12k op/s 2024-03-24T00:43:20.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 -- 172.21.15.123:0/1745448268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8075660 msgr2=0x7fa2c8073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 --2- 172.21.15.123:0/1745448268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8075660 0x7fa2c8073ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009900 tx=0x7fa2b802bcc0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 -- 172.21.15.123:0/1745448268 shutdown_connections 2024-03-24T00:43:20.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 --2- 172.21.15.123:0/1745448268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8109580 0x7fa2c810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 --2- 172.21.15.123:0/1745448268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c80744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 --2- 172.21.15.123:0/1745448268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8075660 0x7fa2c8073ab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.389+0000 7fa2cea00640 1 -- 172.21.15.123:0/1745448268 >> 172.21.15.123:0/1745448268 conn(0x7fa2c80fcc60 msgr2=0x7fa2c80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.390+0000 7fa2cea00640 1 -- 172.21.15.123:0/1745448268 shutdown_connections 2024-03-24T00:43:20.392 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.390+0000 7fa2cea00640 1 -- 172.21.15.123:0/1745448268 wait complete. 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.390+0000 7fa2cea00640 1 Processor -- start 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 -- start start 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 0x7fa2c819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8109580 0x7fa2c81a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c810f580 con 0x7fa2c8075660 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c810f400 con 0x7fa2c8109580 2024-03-24T00:43:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c810f700 con 0x7fa2c8074080 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34400/0 (socket says 172.21.15.123:34400) 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2c7e00640 1 -- 172.21.15.123:0/3399086236 learned_addr learned my addr 172.21.15.123:0/3399086236 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8109580 0x7fa2c81a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.391+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 0x7fa2c819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 -- 172.21.15.123:0/3399086236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8109580 msgr2=0x7fa2c81a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8109580 0x7fa2c81a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 -- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 msgr2=0x7fa2c819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 0x7fa2c819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 -- 172.21.15.123:0/3399086236 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c81a16f0 con 0x7fa2c8074080 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 0x7fa2c819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8000c00 tx=0x7fa2b802c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c5600640 1 -- 172.21.15.123:0/3399086236 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8043020 con 0x7fa2c8074080 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c5600640 1 -- 172.21.15.123:0/3399086236 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b80054e0 con 0x7fa2c8074080 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2c5600640 1 -- 172.21.15.123:0/3399086236 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8003c20 con 0x7fa2c8074080 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.392+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c81a1980 con 0x7fa2c8074080 2024-03-24T00:43:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.393+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c81a1e80 con 0x7fa2c8074080 2024-03-24T00:43:20.404 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.396+0000 7fa2c5600640 1 -- 172.21.15.123:0/3399086236 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa2b8046020 con 0x7fa2c8074080 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.396+0000 7fa2d17a0740 1 Processor -- start 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 -- start start 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 0x5649d59cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cfd40 0x5649d59d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d598d3b0 con 0x5649d59cfd40 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d592f830 con 0x5649d59cabd0 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d58d85d0 con 0x5649d59ca1e0 2024-03-24T00:43:20.405 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34406/0 (socket says 172.21.15.123:34406) 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 -- 172.21.15.123:0/3594192251 learned_addr learned my addr 172.21.15.123:0/3594192251 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cfd40 0x5649d59d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 0x5649d59cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 -- 172.21.15.123:0/3594192251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 msgr2=0x5649d59cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 0x5649d59cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 -- 172.21.15.123:0/3594192251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cfd40 msgr2=0x5649d59d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.406 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.397+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cfd40 0x5649d59d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2cca00640 1 -- 172.21.15.123:0/3594192251 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649d58f02c0 con 0x5649d59ca1e0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 0x5649d59cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2c5600640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 0x7fa2ac0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009fd0 tx=0x7fa2c00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2902472e85160c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2c5600640 1 -- 172.21.15.123:0/3399086236 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2b80f0b80 con 0x7fa2c8074080 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 0x7fa2ac0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2bd200640 1 -- 172.21.15.123:0/3594192251 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0004d20 con 0x5649d59ca1e0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2bd200640 1 -- 172.21.15.123:0/3594192251 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2c0004050 con 0x5649d59ca1e0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.398+0000 7fa2bd200640 1 -- 172.21.15.123:0/3594192251 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0005260 con 0x5649d59ca1e0 2024-03-24T00:43:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3594192251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 msgr2=0x5649d59ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009fd0 tx=0x7fa2c00118d0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2c7e00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 0x7fa2ac0a4d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80bd1d0 tx=0x7fa2b80dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3594192251 shutdown_connections 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cfd40 0x5649d59d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cabd0 0x5649d59cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3594192251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59ca1e0 0x5649d59ca600 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3594192251 >> 172.21.15.123:0/3594192251 conn(0x5649d59c5750 msgr2=0x5649d59c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3594192251 shutdown_connections 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.399+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3594192251 wait complete. 2024-03-24T00:43:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 -- start start 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 Processor -- start 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59db920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d58eba20 con 0x5649d59ca1e0 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d592b8d0 con 0x5649d59cfd40 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.400+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d59d2cf0 con 0x5649d59cabd0 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2d17a0740 10 client.0 _ll_register_callbacks cb 0x5649d59281c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34412/0 (socket says 172.21.15.123:34412) 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3250574935 learned_addr learned my addr 172.21.15.123:0/3250574935 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59db920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 msgr2=0x5649d59db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59db920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3250574935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 msgr2=0x5649d59d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3250574935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649d5644210 con 0x5649d59cabd0 2024-03-24T00:43:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59db920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00126b0 tx=0x7fa2b0012b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a5f53ae276241b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.401+0000 7fa2b5600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004320 con 0x5649d59cabd0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2b0011940 con 0x5649d59cabd0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2c0004050 con 0x5649d59cabd0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2b00044c0 con 0x5649d59cabd0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2d17a0740 10 client.26128 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649d5710620 con 0x5649d59cabd0 2024-03-24T00:43:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2c0004050 con 0x5649d59cabd0 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa2b000f440 con 0x5649d59cabd0 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2d17a0740 10 client.26128 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.402+0000 7fa2d17a0740 10 client.26128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2d17a0740 20 client.26128 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d59e0210 0x5649d59e2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2d17a0740 4 client.26128 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649d593cd70 con 0x5649d59e0210 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d59e0210 0x5649d59e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d59e0210 0x5649d59e2640 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cb34f448d67a53a server_cookie=8010fce8b656639d in_seq=0 out_seq=0 2024-03-24T00:43:20.413 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.403+0000 7fa2b5600640 10 client.26128 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2b5600640 1 -- 172.21.15.123:0/3250574935 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649d593cd70 con 0x5649d59e0210 2024-03-24T00:43:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2b5600640 10 client.26128 handle_command_reply: tid=0 2024-03-24T00:43:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 client.26128 shutdown 2024-03-24T00:43:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 msgr2=0x5649d59d48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00126b0 tx=0x7fa2b0012b70 comp rx=0 tx=0).stop 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 shutdown_connections 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59db920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d59e0210 0x5649d59e2640 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d48e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3250574935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2b5600640 0 client.26128 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 >> 172.21.15.123:0/3250574935 conn(0x5649d59c5750 msgr2=0x5649d59c4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 shutdown_connections 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3250574935 wait complete. 2024-03-24T00:43:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.405+0000 7fa2d17a0740 20 client.26128 trim_cache size 0 max 16384 2024-03-24T00:43:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 Processor -- start 2024-03-24T00:43:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 -- start start 2024-03-24T00:43:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cabd0 0x5649d59db8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cfd40 0x5649d59d48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81a8c00 con 0x5649d59cabd0 2024-03-24T00:43:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81ab1c0 con 0x5649d59ca1e0 2024-03-24T00:43:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.424+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81a9200 con 0x5649d59cfd40 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:34566/0 (socket says 172.21.15.123:34566) 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 -- 172.21.15.123:0/3224016525 learned_addr learned my addr 172.21.15.123:0/3224016525 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cfd40 0x5649d59d48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cabd0 0x5649d59db8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 -- 172.21.15.123:0/3224016525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cfd40 msgr2=0x5649d59d48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cfd40 0x5649d59d48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 -- 172.21.15.123:0/3224016525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cabd0 msgr2=0x5649d59db8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cabd0 0x5649d59db8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 -- 172.21.15.123:0/3224016525 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649d59cb460 con 0x5649d59ca1e0 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.425+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0014040 tx=0x7fa2c0004870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=604c9af90973e200 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.426+0000 7fa2b7e00640 1 -- 172.21.15.123:0/3224016525 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0004050 con 0x5649d59ca1e0 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.426+0000 7fa2b7e00640 1 -- 172.21.15.123:0/3224016525 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2c000f4e0 con 0x5649d59ca1e0 2024-03-24T00:43:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.426+0000 7fa2b7e00640 1 -- 172.21.15.123:0/3224016525 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00046d0 con 0x5649d59ca1e0 2024-03-24T00:43:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3224016525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 msgr2=0x5649d59d43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0014040 tx=0x7fa2c0004870 comp rx=0 tx=0).stop 2024-03-24T00:43:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3224016525 shutdown_connections 2024-03-24T00:43:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cfd40 0x5649d59d48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59cabd0 0x5649d59db8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3224016525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59ca1e0 0x5649d59d43a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3224016525 >> 172.21.15.123:0/3224016525 conn(0x5649d59c5750 msgr2=0x5649d59cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3224016525 shutdown_connections 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3224016525 wait complete. 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.427+0000 7fa2d17a0740 1 -- start start 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 Processor -- start 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59daa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81a9840 con 0x5649d59ca1e0 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81aab60 con 0x5649d59cfd40 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c81a9ea0 con 0x5649d59cabd0 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2d17a0740 10 client.0 _ll_register_callbacks cb 0x7fa2b0009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2cca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:34440/0 (socket says 172.21.15.123:34440) 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.428+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3866880530 learned_addr learned my addr 172.21.15.123:0/3866880530 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 msgr2=0x5649d59daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3866880530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 msgr2=0x5649d59d95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 -- 172.21.15.123:0/3866880530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649d59db080 con 0x5649d59cabd0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2bfe00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5649d59c9e40 tx=0x7fa2b0005ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c173f0b03fb0f9cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2b5600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000fcb0 con 0x5649d59cabd0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2b0005030 con 0x5649d59cabd0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.429+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2c000f4e0 con 0x5649d59cabd0 2024-03-24T00:43:20.433 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2b000fe50 con 0x5649d59cabd0 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 10 client.26134 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649d5a3bf90 con 0x5649d59cabd0 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b003b070 con 0x5649d59cabd0 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa2b0004660 con 0x5649d59cabd0 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 10 client.26134 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 10 client.26134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 20 client.26134 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d5a3c8a0 0x5649d5a3ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 4 client.26134 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5649d59370c0 con 0x5649d5a3c8a0 2024-03-24T00:43:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.430+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d5a3c8a0 0x5649d5a3ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.431+0000 7fa2cca00640 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d5a3c8a0 0x5649d5a3ed00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70ce29901ade6f94 server_cookie=e5cd76f01144ef3b in_seq=0 out_seq=0 2024-03-24T00:43:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:20.431+0000 7fa2b5600640 10 client.26134 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:20.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:20 smithi123 ceph-mon[31487]: pgmap v548: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 35 MiB/s wr, 3.63k op/s 2024-03-24T00:43:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:20 smithi177 ceph-mon[36034]: pgmap v548: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 35 MiB/s wr, 3.63k op/s 2024-03-24T00:43:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:20 smithi204 ceph-mon[36345]: pgmap v548: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 35 MiB/s wr, 3.63k op/s 2024-03-24T00:43:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:22 smithi177 ceph-mon[36034]: pgmap v549: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 40 MiB/s wr, 4.34k op/s 2024-03-24T00:43:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:22 smithi204 ceph-mon[36345]: pgmap v549: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 40 MiB/s wr, 4.34k op/s 2024-03-24T00:43:22.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:22 smithi123 ceph-mon[31487]: pgmap v549: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 40 MiB/s wr, 4.34k op/s 2024-03-24T00:43:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:24 smithi177 ceph-mon[36034]: pgmap v550: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 34 MiB/s wr, 3.74k op/s 2024-03-24T00:43:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:24 smithi204 ceph-mon[36345]: pgmap v550: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 34 MiB/s wr, 3.74k op/s 2024-03-24T00:43:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:24 smithi123 ceph-mon[31487]: pgmap v550: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 34 MiB/s wr, 3.74k op/s 2024-03-24T00:43:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:26 smithi177 ceph-mon[36034]: pgmap v551: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.85k op/s 2024-03-24T00:43:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:26 smithi204 ceph-mon[36345]: pgmap v551: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.85k op/s 2024-03-24T00:43:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:26 smithi123 ceph-mon[31487]: pgmap v551: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.85k op/s 2024-03-24T00:43:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:28 smithi177 ceph-mon[36034]: pgmap v552: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 34 MiB/s wr, 3.66k op/s 2024-03-24T00:43:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:28 smithi204 ceph-mon[36345]: pgmap v552: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 34 MiB/s wr, 3.66k op/s 2024-03-24T00:43:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:28 smithi123 ceph-mon[31487]: pgmap v552: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 34 MiB/s wr, 3.66k op/s 2024-03-24T00:43:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:30 smithi177 ceph-mon[36034]: pgmap v553: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 40 MiB/s wr, 4.49k op/s 2024-03-24T00:43:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:30 smithi204 ceph-mon[36345]: pgmap v553: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 40 MiB/s wr, 4.49k op/s 2024-03-24T00:43:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:30 smithi123 ceph-mon[31487]: pgmap v553: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 40 MiB/s wr, 4.49k op/s 2024-03-24T00:43:31.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:31 smithi177 ceph-mon[36034]: pgmap v554: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 44 MiB/s wr, 5.03k op/s 2024-03-24T00:43:31.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:31 smithi204 ceph-mon[36345]: pgmap v554: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 44 MiB/s wr, 5.03k op/s 2024-03-24T00:43:31.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:31 smithi123 ceph-mon[31487]: pgmap v554: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 44 MiB/s wr, 5.03k op/s 2024-03-24T00:43:34.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:34 smithi123 ceph-mon[31487]: pgmap v555: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 40 MiB/s wr, 4.51k op/s 2024-03-24T00:43:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:34 smithi177 ceph-mon[36034]: pgmap v555: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 40 MiB/s wr, 4.51k op/s 2024-03-24T00:43:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:34 smithi204 ceph-mon[36345]: pgmap v555: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 40 MiB/s wr, 4.51k op/s 2024-03-24T00:43:36.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:36 smithi123 ceph-mon[31487]: pgmap v556: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 42 MiB/s wr, 4.83k op/s 2024-03-24T00:43:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:36 smithi177 ceph-mon[36034]: pgmap v556: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 42 MiB/s wr, 4.83k op/s 2024-03-24T00:43:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:36 smithi204 ceph-mon[36345]: pgmap v556: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 42 MiB/s wr, 4.83k op/s 2024-03-24T00:43:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.226+0000 7fa2b5600640 1 -- 172.21.15.123:0/3866880530 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5649d59370c0 con 0x5649d5a3c8a0 2024-03-24T00:43:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.227+0000 7fa2b5600640 10 client.26134 handle_command_reply: tid=0 2024-03-24T00:43:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.227+0000 7fa2d17a0740 1 client.26134 shutdown 2024-03-24T00:43:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.227+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 msgr2=0x5649d59d9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.227+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5649d59c9e40 tx=0x7fa2b0005ca0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 shutdown_connections 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649d59cfd40 0x5649d59daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5649d5a3c8a0 0x5649d5a3ed00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2b5600640 0 client.26134 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5649d59cabd0 0x5649d59d9af0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 --2- 172.21.15.123:0/3866880530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5649d59ca1e0 0x5649d59d95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 >> 172.21.15.123:0/3866880530 conn(0x5649d59c5750 msgr2=0x5649d59d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 shutdown_connections 2024-03-24T00:43:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.228+0000 7fa2d17a0740 1 -- 172.21.15.123:0/3866880530 wait complete. 2024-03-24T00:43:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.229+0000 7fa2d17a0740 20 client.26134 trim_cache size 0 max 16384 2024-03-24T00:43:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:43:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:43:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:43:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:43:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 msgr2=0x7fa2ac0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 0x7fa2ac0a4d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80bd1d0 tx=0x7fa2b80dd040 comp rx=0 tx=0).stop 2024-03-24T00:43:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 msgr2=0x7fa2c8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8000c00 tx=0x7fa2b802c6e0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 shutdown_connections 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2c8109580 0x7fa2c81a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa2ac0a28a0 0x7fa2ac0a4d60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2c8075660 0x7fa2c819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 --2- 172.21.15.123:0/3399086236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c8074080 0x7fa2c8199ad0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 >> 172.21.15.123:0/3399086236 conn(0x7fa2c80fcc60 msgr2=0x7fa2c8105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 shutdown_connections 2024-03-24T00:43:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:38.230+0000 7fa2cea00640 1 -- 172.21.15.123:0/3399086236 wait complete. 2024-03-24T00:43:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:38 smithi177 ceph-mon[36034]: pgmap v557: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 36 MiB/s wr, 4.24k op/s 2024-03-24T00:43:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:38 smithi204 ceph-mon[36345]: pgmap v557: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 36 MiB/s wr, 4.24k op/s 2024-03-24T00:43:38.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:38 smithi123 ceph-mon[31487]: pgmap v557: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 36 MiB/s wr, 4.24k op/s 2024-03-24T00:43:39.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:43:39.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.935+0000 7fdffac00640 1 -- 172.21.15.123:0/2023004246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff4109430 msgr2=0x7fdff410b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.935+0000 7fdffac00640 1 --2- 172.21.15.123:0/2023004246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff4109430 0x7fdff410b8d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009ca0 tx=0x7fdfec02c180 comp rx=0 tx=0).stop 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 -- 172.21.15.123:0/2023004246 shutdown_connections 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 --2- 172.21.15.123:0/2023004246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff4109430 0x7fdff410b8d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 --2- 172.21.15.123:0/2023004246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff40740a0 0x7fdff4074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 --2- 172.21.15.123:0/2023004246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 0x7fdff4073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 -- 172.21.15.123:0/2023004246 >> 172.21.15.123:0/2023004246 conn(0x7fdff40fcb10 msgr2=0x7fdff40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 -- 172.21.15.123:0/2023004246 shutdown_connections 2024-03-24T00:43:39.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.936+0000 7fdffac00640 1 -- 172.21.15.123:0/2023004246 wait complete. 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 Processor -- start 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 -- start start 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 0x7fdff4104e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 0x7fdff4101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff410f420 con 0x7fdff4075680 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff410f2a0 con 0x7fdff40740a0 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffac00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff410f5a0 con 0x7fdff4109430 2024-03-24T00:43:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdff9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdff9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39102/0 (socket says 172.21.15.123:39102) 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdff9800640 1 -- 172.21.15.123:0/1790860270 learned_addr learned my addr 172.21.15.123:0/1790860270 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdffa200640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 0x7fdff4101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.937+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 0x7fdff4104e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 -- 172.21.15.123:0/1790860270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 msgr2=0x7fdff4101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 0x7fdff4101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 -- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 msgr2=0x7fdff4104e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 0x7fdff4104e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 -- 172.21.15.123:0/1790860270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4101d50 con 0x7fdff40740a0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdffa200640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 0x7fdff4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff9800640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800e9f0 tx=0x7fdfe800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff2000640 1 -- 172.21.15.123:0/1790860270 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe800cd80 con 0x7fdff40740a0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff2000640 1 -- 172.21.15.123:0/1790860270 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfe8010910 con 0x7fdff40740a0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdff2000640 1 -- 172.21.15.123:0/1790860270 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8018990 con 0x7fdff40740a0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.938+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff41a5e60 con 0x7fdff40740a0 2024-03-24T00:43:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.939+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff41a6210 con 0x7fdff40740a0 2024-03-24T00:43:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.942+0000 7fdff2000640 1 -- 172.21.15.123:0/1790860270 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdfe8010ac0 con 0x7fdff40740a0 2024-03-24T00:43:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.942+0000 7fdfff031740 1 Processor -- start 2024-03-24T00:43:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.942+0000 7fdfff031740 1 -- start start 2024-03-24T00:43:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.942+0000 7fdfff031740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdfff031740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6a40 0x55e66d2cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdfff031740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2cbbb0 0x55e66d2ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdfff031740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d22b250 con 0x55e66d2c6a40 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdfff031740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d1d3a30 con 0x55e66d2cbbb0 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdfff031740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2272f0 con 0x55e66d2c6050 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2cbbb0 0x55e66d2ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdff9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6a40 0x55e66d2cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37016/0 (socket says 172.21.15.123:37016) 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 -- 172.21.15.123:0/1845302434 learned_addr learned my addr 172.21.15.123:0/1845302434 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 -- 172.21.15.123:0/1845302434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2cbbb0 msgr2=0x55e66d2ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2cbbb0 0x55e66d2ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 -- 172.21.15.123:0/1845302434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6a40 msgr2=0x55e66d2cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6a40 0x55e66d2cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.943+0000 7fdffa200640 1 -- 172.21.15.123:0/1845302434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e66d1be690 con 0x55e66d2c6050 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdffa200640 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdfec02c790 tx=0x7fdfec005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb114f54ee9d413d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdff2000640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 0x7fdfdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdfe6a00640 1 -- 172.21.15.123:0/1845302434 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec043070 con 0x55e66d2c6050 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdff2000640 1 -- 172.21.15.123:0/1790860270 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdfe80ced40 con 0x7fdff40740a0 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdfe6a00640 1 -- 172.21.15.123:0/1845302434 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfec003e50 con 0x55e66d2c6050 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdfe6a00640 1 -- 172.21.15.123:0/1845302434 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec039040 con 0x55e66d2c6050 2024-03-24T00:43:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.944+0000 7fdff9800640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 0x7fdfdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 -- 172.21.15.123:0/1845302434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 msgr2=0x55e66d2c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdfec02c790 tx=0x7fdfec005f80 comp rx=0 tx=0).stop 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdff9800640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 0x7fdfdc0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdfe809b6d0 tx=0x7fdfe80d2780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 -- 172.21.15.123:0/1845302434 shutdown_connections 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2cbbb0 0x55e66d2ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6a40 0x55e66d2cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 --2- 172.21.15.123:0/1845302434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2c6470 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 -- 172.21.15.123:0/1845302434 >> 172.21.15.123:0/1845302434 conn(0x55e66d28e200 msgr2=0x55e66d2c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 -- 172.21.15.123:0/1845302434 shutdown_connections 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.945+0000 7fdfff031740 1 -- 172.21.15.123:0/1845302434 wait complete. 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 -- start start 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 Processor -- start 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 0x55e66d2d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2ceb00 con 0x55e66d2c6050 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d1e6e80 con 0x55e66d2c6a40 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2cec80 con 0x55e66d2cbbb0 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.946+0000 7fdfff031740 10 client.0 _ll_register_callbacks cb 0x55e66d1f9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdffa200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 0x55e66d2d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdffa200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 0x55e66d2d0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55694/0 (socket says 172.21.15.123:55694) 2024-03-24T00:43:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdffa200640 1 -- 172.21.15.123:0/1059547831 learned_addr learned my addr 172.21.15.123:0/1059547831 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdffa200640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 -- 172.21.15.123:0/1059547831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 msgr2=0x55e66d2d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 -- 172.21.15.123:0/1059547831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 msgr2=0x55e66d2d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 0x55e66d2d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 -- 172.21.15.123:0/1059547831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e66d0097d0 con 0x55e66d2c6a40 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdffa200640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.947+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d0540 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdff41028a0 tx=0x7fdfe0012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae551c29f21cf34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfdb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0004020 con 0x55e66d2c6a40 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdfe00041c0 con 0x55e66d2c6a40 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfec003e50 con 0x55e66d2c6a40 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfe00043c0 con 0x55e66d2c6a40 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfff031740 10 client.26135 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:39.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e66cf3f3f0 con 0x55e66d2c6a40 2024-03-24T00:43:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.948+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfec003e50 con 0x55e66d2c6a40 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfe0003bf0 con 0x55e66d2c6a40 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 10 client.26135 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 10 client.26135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 20 client.26135 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d2dbde0 0x55e66d2de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 4 client.26135 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e66d238790 con 0x55e66d2dbde0 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.949+0000 7fdffa200640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d2dbde0 0x55e66d2de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.950+0000 7fdffa200640 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d2dbde0 0x55e66d2de240 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c314afcdfa18fe4 server_cookie=1e392363f9a0c78f in_seq=0 out_seq=0 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.950+0000 7fdfdb400640 10 client.26135 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.950+0000 7fdfdb400640 1 -- 172.21.15.123:0/1059547831 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e66d238790 con 0x55e66d2dbde0 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.950+0000 7fdfdb400640 10 client.26135 handle_command_reply: tid=0 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 client.26135 shutdown 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 msgr2=0x55e66d2d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d0540 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdff41028a0 tx=0x7fdfe0012b60 comp rx=0 tx=0).stop 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 shutdown_connections 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d2dbde0 0x55e66d2de240 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d0540 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfdb400640 0 client.26135 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:39.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 --2- 172.21.15.123:0/1059547831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2c6050 0x55e66d2d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 >> 172.21.15.123:0/1059547831 conn(0x55e66d28e200 msgr2=0x55e66d2c0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 shutdown_connections 2024-03-24T00:43:39.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 1 -- 172.21.15.123:0/1059547831 wait complete. 2024-03-24T00:43:39.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.951+0000 7fdfff031740 20 client.26135 trim_cache size 0 max 16384 2024-03-24T00:43:39.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.969+0000 7fdfff031740 1 Processor -- start 2024-03-24T00:43:39.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.969+0000 7fdfff031740 1 -- start start 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 0x55e66d2cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41aceb0 con 0x55e66d2cbbb0 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41af470 con 0x55e66d2c6050 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdfff031740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41ad4b0 con 0x55e66d2c6a40 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37042/0 (socket says 172.21.15.123:37042) 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 -- 172.21.15.123:0/4058150930 learned_addr learned my addr 172.21.15.123:0/4058150930 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdffa200640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 0x55e66d2cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdffa200640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 -- 172.21.15.123:0/4058150930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 msgr2=0x55e66d2cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 0x55e66d2cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 -- 172.21.15.123:0/4058150930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 msgr2=0x55e66d2c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdff8e00640 1 -- 172.21.15.123:0/4058150930 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e66ceff0f0 con 0x55e66d2c6a40 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.970+0000 7fdffa200640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 0x55e66d2cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.971+0000 7fdff8e00640 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00059f0 tx=0x7fdfe0009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9054c6bf1a378fcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.971+0000 7fdfe6000640 1 -- 172.21.15.123:0/4058150930 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0003fe0 con 0x55e66d2c6a40 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.971+0000 7fdfe6000640 1 -- 172.21.15.123:0/4058150930 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfe0004180 con 0x55e66d2c6a40 2024-03-24T00:43:39.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.971+0000 7fdfe6000640 1 -- 172.21.15.123:0/4058150930 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe00035f0 con 0x55e66d2c6a40 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- 172.21.15.123:0/4058150930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 msgr2=0x55e66d2d7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00059f0 tx=0x7fdfe0009830 comp rx=0 tx=0).stop 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- 172.21.15.123:0/4058150930 shutdown_connections 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6a40 0x55e66d2d7540 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 --2- 172.21.15.123:0/4058150930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6050 0x55e66d2cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- 172.21.15.123:0/4058150930 >> 172.21.15.123:0/4058150930 conn(0x55e66d28e200 msgr2=0x55e66d2c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- 172.21.15.123:0/4058150930 shutdown_connections 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- 172.21.15.123:0/4058150930 wait complete. 2024-03-24T00:43:39.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 -- start start 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.972+0000 7fdfff031740 1 Processor -- start 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41adaf0 con 0x55e66d2cbbb0 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41aee10 con 0x55e66d2c6a40 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41ae150 con 0x55e66d2c6050 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdfff031740 10 client.0 _ll_register_callbacks cb 0x7fdfe0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdffa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdffa200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdffa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37052/0 (socket says 172.21.15.123:37052) 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdffa200640 1 -- 172.21.15.123:0/1432511594 learned_addr learned my addr 172.21.15.123:0/1432511594 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdff8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39140/0 (socket says 172.21.15.123:39140) 2024-03-24T00:43:39.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.973+0000 7fdffa200640 1 -- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 msgr2=0x55e66d2d56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdffa200640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdffa200640 1 -- 172.21.15.123:0/1432511594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 msgr2=0x55e66d2d5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdffa200640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdffa200640 1 -- 172.21.15.123:0/1432511594 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e66d2d6170 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdffa200640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0096c0 tx=0x7fdfec02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b43f164162a2b983 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfdb400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec039040 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdfec003970 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfe0004180 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfec043040 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfff031740 10 client.26152 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.974+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e66d3383d0 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfe0004180 con 0x55e66d2c6050 2024-03-24T00:43:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfec0556a0 con 0x55e66d2c6050 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 10 client.26152 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 20 client.26152 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d338880 0x55e66d33ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 4 client.26152 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e66d232ae0 con 0x55e66d338880 2024-03-24T00:43:39.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.975+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d338880 0x55e66d33ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:39.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdff8e00640 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d338880 0x55e66d33ac70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3411c3f52469bc87 server_cookie=15e43f3c20565373 in_seq=0 out_seq=0 2024-03-24T00:43:39.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfdb400640 10 client.26152 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:39.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfdb400640 1 -- 172.21.15.123:0/1432511594 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e66d232ae0 con 0x55e66d338880 2024-03-24T00:43:39.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfdb400640 10 client.26152 handle_command_reply: tid=0 2024-03-24T00:43:39.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 client.26152 shutdown 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 msgr2=0x55e66d2d70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0096c0 tx=0x7fdfec02e040 comp rx=0 tx=0).stop 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 shutdown_connections 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e66d2cbbb0 0x55e66d2d5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e66d2c6a40 0x55e66d2d56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55e66d338880 0x55e66d33ac70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 --2- 172.21.15.123:0/1432511594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e66d2c6050 0x55e66d2d70e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfdb400640 0 client.26152 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 >> 172.21.15.123:0/1432511594 conn(0x55e66d28e200 msgr2=0x55e66d2c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 shutdown_connections 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 1 -- 172.21.15.123:0/1432511594 wait complete. 2024-03-24T00:43:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.976+0000 7fdfff031740 20 client.26152 trim_cache size 0 max 16384 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 msgr2=0x7fdfdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 0x7fdfdc0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdfe809b6d0 tx=0x7fdfe80d2780 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 msgr2=0x7fdff4104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800e9f0 tx=0x7fdfe800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 shutdown_connections 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdff4109430 0x7fdff4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdfdc0a2880 0x7fdfdc0a4d40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff4075680 0x7fdff4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 --2- 172.21.15.123:0/1790860270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdff40740a0 0x7fdff4104920 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 >> 172.21.15.123:0/1790860270 conn(0x7fdff40fcb10 msgr2=0x7fdff41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 shutdown_connections 2024-03-24T00:43:39.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:39.977+0000 7fdffac00640 1 -- 172.21.15.123:0/1790860270 wait complete. 2024-03-24T00:43:39.994 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:43:39.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 -- 172.21.15.123:0/150378934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1015e0 msgr2=0x7f078c101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 --2- 172.21.15.123:0/150378934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1015e0 0x7f078c101a60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0778009860 tx=0x7f077802bb70 comp rx=0 tx=0).stop 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 -- 172.21.15.123:0/150378934 shutdown_connections 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 --2- 172.21.15.123:0/150378934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c101fa0 0x7f078c108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 --2- 172.21.15.123:0/150378934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1015e0 0x7f078c101a60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 --2- 172.21.15.123:0/150378934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c1003f0 0x7f078c100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.201+0000 7f078be00640 1 -- 172.21.15.123:0/150378934 >> 172.21.15.123:0/150378934 conn(0x7f078c0fb980 msgr2=0x7f078c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.202+0000 7f078be00640 1 -- 172.21.15.123:0/150378934 shutdown_connections 2024-03-24T00:43:40.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.202+0000 7f078be00640 1 -- 172.21.15.123:0/150378934 wait complete. 2024-03-24T00:43:40.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 Processor -- start 2024-03-24T00:43:40.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 -- start start 2024-03-24T00:43:40.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 0x7f078c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c101fa0 0x7f078c1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c10c080 con 0x7f078c1015e0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c10bf00 con 0x7f078c1003f0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c10c200 con 0x7f078c101fa0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39160/0 (socket says 172.21.15.123:39160) 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078aa00640 1 -- 172.21.15.123:0/163139482 learned_addr learned my addr 172.21.15.123:0/163139482 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078a000640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 0x7f078c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.203+0000 7f078b400640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c101fa0 0x7f078c1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 -- 172.21.15.123:0/163139482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c101fa0 msgr2=0x7f078c1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c101fa0 0x7f078c1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 -- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 msgr2=0x7f078c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 0x7f078c199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 -- 172.21.15.123:0/163139482 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c1a1630 con 0x7f078c1003f0 2024-03-24T00:43:40.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078a000640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 0x7f078c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078aa00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f077c00e9a0 tx=0x7f077c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c00cd80 con 0x7f078c1003f0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f077c010910 con 0x7f078c1003f0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c1a1920 con 0x7f078c1003f0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c018a30 con 0x7f078c1003f0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.204+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c1a1e90 con 0x7f078c1003f0 2024-03-24T00:43:40.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.206+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c0683b0 con 0x7f078c1003f0 2024-03-24T00:43:40.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.207+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f077c010ac0 con 0x7f078c1003f0 2024-03-24T00:43:40.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.209+0000 7f0783400640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 0x7f07700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.209+0000 7f078a000640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 0x7f07700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.209+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f077c0cee40 con 0x7f078c1003f0 2024-03-24T00:43:40.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.210+0000 7f078a000640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 0x7f07700a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0778009830 tx=0x7f0778005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.213+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f077c053740 con 0x7f078c1003f0 2024-03-24T00:43:40.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.524+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f078c1a2120 con 0x7f078c1003f0 2024-03-24T00:43:40.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.526+0000 7f0783400640 1 -- 172.21.15.123:0/163139482 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f077c09be90 con 0x7f078c1003f0 2024-03-24T00:43:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:43:40.530 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:43:40.530 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 msgr2=0x7f07700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 0x7f07700a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0778009830 tx=0x7f0778005d20 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 msgr2=0x7f078c199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f077c00e9a0 tx=0x7f077c00ee60 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 shutdown_connections 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c101fa0 0x7f078c1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f07700a2880 0x7f07700a4d40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c1015e0 0x7f078c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 --2- 172.21.15.123:0/163139482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f078c1003f0 0x7f078c199a10 secure :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f077c00e9a0 tx=0x7f077c00ee60 comp rx=0 tx=0).stop 2024-03-24T00:43:40.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.535+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 >> 172.21.15.123:0/163139482 conn(0x7f078c0fb980 msgr2=0x7f078c0fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.538+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 shutdown_connections 2024-03-24T00:43:40.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.538+0000 7f078be00640 1 -- 172.21.15.123:0/163139482 wait complete. 2024-03-24T00:43:40.574 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:43:40.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:43:40.703 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:40 smithi123 ceph-mon[31487]: pgmap v558: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 4.51k op/s 2024-03-24T00:43:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:40 smithi177 ceph-mon[36034]: pgmap v558: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 4.51k op/s 2024-03-24T00:43:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:40 smithi204 ceph-mon[36345]: pgmap v558: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 4.51k op/s 2024-03-24T00:43:40.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 -- 172.21.15.123:0/1334345618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654102000 msgr2=0x7fd6541043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 --2- 172.21.15.123:0/1334345618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654102000 0x7fd6541043f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd648009900 tx=0x7fd64802bc50 comp rx=0 tx=0).stop 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 -- 172.21.15.123:0/1334345618 shutdown_connections 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 --2- 172.21.15.123:0/1334345618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654107260 0x7fd6541096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 --2- 172.21.15.123:0/1334345618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654104930 0x7fd654106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 --2- 172.21.15.123:0/1334345618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654102000 0x7fd6541043f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 -- 172.21.15.123:0/1334345618 >> 172.21.15.123:0/1334345618 conn(0x7fd6540fba10 msgr2=0x7fd6540fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 -- 172.21.15.123:0/1334345618 shutdown_connections 2024-03-24T00:43:40.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.858+0000 7fd659e00640 1 -- 172.21.15.123:0/1334345618 wait complete. 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.859+0000 7fd659e00640 1 Processor -- start 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.859+0000 7fd659e00640 1 -- start start 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.859+0000 7fd659e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.859+0000 7fd659e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654104930 0x7fd6540fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.859+0000 7fd659e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410d260 con 0x7fd654102000 2024-03-24T00:43:40.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410d0e0 con 0x7fd654107260 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410d3e0 con 0x7fd654104930 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55730/0 (socket says 172.21.15.123:55730) 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 -- 172.21.15.123:0/208757076 learned_addr learned my addr 172.21.15.123:0/208757076 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39186/0 (socket says 172.21.15.123:39186) 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd653e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654104930 0x7fd6540fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 -- 172.21.15.123:0/208757076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654104930 msgr2=0x7fd6540fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654104930 0x7fd6540fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 -- 172.21.15.123:0/208757076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 msgr2=0x7fd6541004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd658a00640 1 -- 172.21.15.123:0/208757076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd654100a30 con 0x7fd654102000 2024-03-24T00:43:40.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.860+0000 7fd659400640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd658a00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd64802c260 tx=0x7fd64802c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd651600640 1 -- 172.21.15.123:0/208757076 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648041070 con 0x7fd654102000 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd651600640 1 -- 172.21.15.123:0/208757076 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd648004450 con 0x7fd654102000 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd654100cc0 con 0x7fd654102000 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd651600640 1 -- 172.21.15.123:0/208757076 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648034440 con 0x7fd654102000 2024-03-24T00:43:40.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.861+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6541a6310 con 0x7fd654102000 2024-03-24T00:43:40.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.864+0000 7fd65e168740 1 Processor -- start 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.864+0000 7fd651600640 1 -- 172.21.15.123:0/208757076 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd648044070 con 0x7fd654102000 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 -- start start 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c5322c0 con 0x55f04c5cd9f0 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c4dac90 con 0x55f04c5cf580 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd65e168740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c52e360 con 0x55f04c5cebc0 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd658a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd659400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd653e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd659400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55738/0 (socket says 172.21.15.123:55738) 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd659400640 1 -- 172.21.15.123:0/2407186552 learned_addr learned my addr 172.21.15.123:0/2407186552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.865+0000 7fd658a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37106/0 (socket says 172.21.15.123:37106) 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 -- 172.21.15.123:0/2407186552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 msgr2=0x55f04c5cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 -- 172.21.15.123:0/2407186552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 msgr2=0x55f04c5cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 -- 172.21.15.123:0/2407186552 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f04c4ff080 con 0x55f04c5cf580 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd659400640 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:40.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd653e00640 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5d5e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd654069360 tx=0x7fd640011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255b55f589642315 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd645c00640 1 -- 172.21.15.123:0/2407186552 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640004050 con 0x55f04c5cf580 2024-03-24T00:43:40.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd645c00640 1 -- 172.21.15.123:0/2407186552 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd640005500 con 0x55f04c5cf580 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd645c00640 1 -- 172.21.15.123:0/2407186552 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640004db0 con 0x55f04c5cf580 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.866+0000 7fd651600640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 0x7fd63c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd651600640 1 -- 172.21.15.123:0/208757076 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd6480f0520 con 0x7fd654102000 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd659400640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 0x7fd63c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 -- 172.21.15.123:0/2407186552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 msgr2=0x55f04c5d5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5d5e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd654069360 tx=0x7fd640011810 comp rx=0 tx=0).stop 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 -- 172.21.15.123:0/2407186552 shutdown_connections 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5d5e10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 --2- 172.21.15.123:0/2407186552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 -- 172.21.15.123:0/2407186552 >> 172.21.15.123:0/2407186552 conn(0x55f04c5c8de0 msgr2=0x55f04c5c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 -- 172.21.15.123:0/2407186552 shutdown_connections 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd659400640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 0x7fd63c0a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55f04c5ceac0 tx=0x7fd64c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.867+0000 7fd65e168740 1 -- 172.21.15.123:0/2407186552 wait complete. 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 -- start start 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 Processor -- start 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5d8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.868+0000 7fd65e168740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c5d6590 con 0x55f04c5cd9f0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd65e168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c4ee0d0 con 0x55f04c5cf580 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd65e168740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04c5d6710 con 0x55f04c5cebc0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd65e168740 10 client.0 _ll_register_callbacks cb 0x55f04c4f2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd653e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd658a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39196/0 (socket says 172.21.15.123:39196) 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd653e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5d8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37120/0 (socket says 172.21.15.123:37120) 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 -- 172.21.15.123:0/1350845815 learned_addr learned my addr 172.21.15.123:0/1350845815 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 -- 172.21.15.123:0/1350845815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 msgr2=0x55f04c5d8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5d8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 -- 172.21.15.123:0/1350845815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 msgr2=0x55f04c5d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5d7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd659400640 1 -- 172.21.15.123:0/1350845815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f04c2463f0 con 0x55f04c5cf580 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.869+0000 7fd658a00640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd659400640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0047a0 tx=0x7fd64c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755ffc258bc7d3fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd63a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c02d060 con 0x55f04c5cf580 2024-03-24T00:43:40.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd64c017030 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd640005500 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd64c0047d0 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd65e168740 10 client.26159 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f04c3107d0 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.870+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd640005500 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd64c028950 con 0x55f04c5cf580 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 10 client.26159 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 10 client.26159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 20 client.26159 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c5e3740 0x55f04c5e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 4 client.26159 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f04c53f800 con 0x55f04c5e3740 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.871+0000 7fd658a00640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c5e3740 0x55f04c5e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd63a000640 10 client.26159 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:40.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd658a00640 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c5e3740 0x55f04c5e5be0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38506079689accaf server_cookie=7bdf1057f3d988be in_seq=0 out_seq=0 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd63a000640 1 -- 172.21.15.123:0/1350845815 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f04c53f800 con 0x55f04c5e3740 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd63a000640 10 client.26159 handle_command_reply: tid=0 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd65e168740 1 client.26159 shutdown 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 msgr2=0x55f04c5df1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.872+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0047a0 tx=0x7fd64c004ab0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 shutdown_connections 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5df1c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c5e3740 0x55f04c5e5be0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5d8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd63a000640 0 client.26159 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 --2- 172.21.15.123:0/1350845815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 >> 172.21.15.123:0/1350845815 conn(0x55f04c5c8de0 msgr2=0x55f04c5c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 shutdown_connections 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 1 -- 172.21.15.123:0/1350845815 wait complete. 2024-03-24T00:43:40.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.873+0000 7fd65e168740 20 client.26159 trim_cache size 0 max 16384 2024-03-24T00:43:40.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 Processor -- start 2024-03-24T00:43:40.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 -- start start 2024-03-24T00:43:40.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5df180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541acf00 con 0x55f04c5cd9f0 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541af4c0 con 0x55f04c5cf580 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.887+0000 7fd65e168740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541ad500 con 0x55f04c5cebc0 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd658a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39212/0 (socket says 172.21.15.123:39212) 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 -- 172.21.15.123:0/3651608563 learned_addr learned my addr 172.21.15.123:0/3651608563 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd653e00640 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5df180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 -- 172.21.15.123:0/3651608563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 msgr2=0x55f04c5df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5df180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 -- 172.21.15.123:0/3651608563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 msgr2=0x55f04c5c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 -- 172.21.15.123:0/3651608563 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f04c5ce640 con 0x55f04c5cf580 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd659400640 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd64c018040 tx=0x7fd64c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=925a578308e7b688 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.888+0000 7fd645200640 1 -- 172.21.15.123:0/3651608563 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c038030 con 0x55f04c5cf580 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd645200640 1 -- 172.21.15.123:0/3651608563 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64c03a050 con 0x55f04c5cf580 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd645200640 1 -- 172.21.15.123:0/3651608563 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c028420 con 0x55f04c5cf580 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 -- 172.21.15.123:0/3651608563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 msgr2=0x55f04c5ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd64c018040 tx=0x7fd64c009f90 comp rx=0 tx=0).stop 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 -- 172.21.15.123:0/3651608563 shutdown_connections 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cf580 0x55f04c5ea4a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cebc0 0x55f04c5df180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 --2- 172.21.15.123:0/3651608563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cd9f0 0x55f04c5c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.889+0000 7fd65e168740 1 -- 172.21.15.123:0/3651608563 >> 172.21.15.123:0/3651608563 conn(0x55f04c5c8de0 msgr2=0x55f04c5c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- 172.21.15.123:0/3651608563 shutdown_connections 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- 172.21.15.123:0/3651608563 wait complete. 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- start start 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 Processor -- start 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 0x55f04c5dec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cf580 0x55f04c5dd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541adb40 con 0x55f04c5cf580 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541aee60 con 0x55f04c5cebc0 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.890+0000 7fd65e168740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541ae1a0 con 0x55f04c5cd9f0 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd65e168740 10 client.0 _ll_register_callbacks cb 0x7fd64c003c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd658a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 0x55f04c5dec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39226/0 (socket says 172.21.15.123:39226) 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 -- 172.21.15.123:0/2638446789 learned_addr learned my addr 172.21.15.123:0/2638446789 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:40.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd659400640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cf580 0x55f04c5dd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 -- 172.21.15.123:0/2638446789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 msgr2=0x55f04c5dec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 0x55f04c5dec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 -- 172.21.15.123:0/2638446789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cf580 msgr2=0x55f04c5dd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cf580 0x55f04c5dd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 -- 172.21.15.123:0/2638446789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f04c5ddac0 con 0x55f04c5cebc0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd658a00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 0x55f04c5dec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd653e00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55f04c5c9d50 tx=0x7fd640004280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1346ab41bc109c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.891+0000 7fd63a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640005450 con 0x55f04c5cebc0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd6400055f0 con 0x55f04c5cebc0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd64c03a050 con 0x55f04c5cebc0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd640009810 con 0x55f04c5cebc0 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd65e168740 10 client.26165 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:40.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f04c63fbf0 con 0x55f04c5cebc0 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64c03a050 con 0x55f04c5cebc0 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd64002b020 con 0x55f04c5cebc0 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd65e168740 10 client.26165 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.892+0000 7fd65e168740 10 client.26165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd65e168740 20 client.26165 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c6403b0 0x55f04c6427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd65e168740 4 client.26165 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f04c539b50 con 0x55f04c6403b0 2024-03-24T00:43:40.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd658a00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c6403b0 0x55f04c6427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:40.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd63a000640 10 client.26165 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:40.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.893+0000 7fd658a00640 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c6403b0 0x55f04c6427d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60483bc14b76b3cb server_cookie=1d5c37292ce28e44 in_seq=0 out_seq=0 2024-03-24T00:43:40.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd63a000640 1 -- 172.21.15.123:0/2638446789 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f04c539b50 con 0x55f04c6403b0 2024-03-24T00:43:40.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd63a000640 10 client.26165 handle_command_reply: tid=0 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 client.26165 shutdown 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 msgr2=0x55f04c5df160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55f04c5c9d50 tx=0x7fd640004280 comp rx=0 tx=0).stop 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 shutdown_connections 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f04c5cf580 0x55f04c5dd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f04c5cebc0 0x55f04c5df160 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f04c6403b0 0x55f04c6427d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 --2- 172.21.15.123:0/2638446789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f04c5cd9f0 0x55f04c5dec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd63a000640 0 client.26165 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 >> 172.21.15.123:0/2638446789 conn(0x55f04c5c8de0 msgr2=0x55f04c5c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 shutdown_connections 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 1 -- 172.21.15.123:0/2638446789 wait complete. 2024-03-24T00:43:40.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.894+0000 7fd65e168740 20 client.26165 trim_cache size 0 max 16384 2024-03-24T00:43:40.899 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:43:40.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 msgr2=0x7fd63c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 0x7fd63c0a4d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55f04c5ceac0 tx=0x7fd64c009500 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 msgr2=0x7fd654101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd64802c260 tx=0x7fd64802c670 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 shutdown_connections 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd654107260 0x7fd6541004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd654104930 0x7fd6540fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd63c0a2880 0x7fd63c0a4d40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 --2- 172.21.15.123:0/208757076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd654102000 0x7fd654101970 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 >> 172.21.15.123:0/208757076 conn(0x7fd6540fba10 msgr2=0x7fd6541028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 shutdown_connections 2024-03-24T00:43:40.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:40.895+0000 7fd659e00640 1 -- 172.21.15.123:0/208757076 wait complete. 2024-03-24T00:43:41.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:41 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/163139482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:41 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/163139482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:41.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:41 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/163139482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:43:41.913 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:43:41.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:41.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:43:42.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.238+0000 7f1ed5e00640 1 -- 172.21.15.123:0/811506512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed010dd70 msgr2=0x7f1ed0110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.238+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/811506512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed010dd70 0x7f1ed0110230 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009370 tx=0x7f1ebc02bd50 comp rx=0 tx=0).stop 2024-03-24T00:43:42.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 -- 172.21.15.123:0/811506512 shutdown_connections 2024-03-24T00:43:42.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/811506512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed010dd70 0x7f1ed0110230 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/811506512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed010d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/811506512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed0108910 0x7f1ed010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 -- 172.21.15.123:0/811506512 >> 172.21.15.123:0/811506512 conn(0x7f1ed006e500 msgr2=0x7f1ed0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 -- 172.21.15.123:0/811506512 shutdown_connections 2024-03-24T00:43:42.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.239+0000 7f1ed5e00640 1 -- 172.21.15.123:0/811506512 wait complete. 2024-03-24T00:43:42.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.241+0000 7f1ed5e00640 1 Processor -- start 2024-03-24T00:43:42.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.242+0000 7f1ed5e00640 1 -- start start 2024-03-24T00:43:42.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.243+0000 7f1ed5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.243+0000 7f1ed5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ed5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 0x7f1ed01add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ece000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ece000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37152/0 (socket says 172.21.15.123:37152) 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ed5e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed01139a0 con 0x7f1ed010dd70 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ed5e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0113820 con 0x7f1ed0108910 2024-03-24T00:43:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ed5e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0113b20 con 0x7f1ed010b2d0 2024-03-24T00:43:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ecea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ecea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39242/0 (socket says 172.21.15.123:39242) 2024-03-24T00:43:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.244+0000 7f1ecea00640 1 -- 172.21.15.123:0/4143560397 learned_addr learned my addr 172.21.15.123:0/4143560397 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.245+0000 7f1ecf400640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 0x7f1ed01add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ece000640 1 -- 172.21.15.123:0/4143560397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 msgr2=0x7f1ed01a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ece000640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ece000640 1 -- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 msgr2=0x7f1ed01add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ece000640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 0x7f1ed01add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ece000640 1 -- 172.21.15.123:0/4143560397 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed01ae390 con 0x7f1ed010b2d0 2024-03-24T00:43:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ecf400640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 0x7f1ed01add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.246+0000 7f1ecea00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.247+0000 7f1ece000640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00737f0 tx=0x7f1ec0009bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.252+0000 7f1ed4a00640 1 -- 172.21.15.123:0/4143560397 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec001b070 con 0x7f1ed010b2d0 2024-03-24T00:43:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.252+0000 7f1ed4a00640 1 -- 172.21.15.123:0/4143560397 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ec0009e10 con 0x7f1ed010b2d0 2024-03-24T00:43:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.252+0000 7f1ed4a00640 1 -- 172.21.15.123:0/4143560397 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00153b0 con 0x7f1ed010b2d0 2024-03-24T00:43:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.252+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed01ae680 con 0x7f1ed010b2d0 2024-03-24T00:43:42.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.252+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed01aeb80 con 0x7f1ed010b2d0 2024-03-24T00:43:42.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.258+0000 7f1ed8bfb740 1 Processor -- start 2024-03-24T00:43:42.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.258+0000 7f1ed8bfb740 1 -- start start 2024-03-24T00:43:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 0x55c675a8a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed4a00640 1 -- 172.21.15.123:0/4143560397 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1ec000ca60 con 0x7f1ed010b2d0 2024-03-24T00:43:42.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 0x55c675a8f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8fc00 0x55c675a92090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6759efeb0 con 0x55c675a8a0a0 2024-03-24T00:43:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c675998a60 con 0x55c675a8fc00 2024-03-24T00:43:42.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.261+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6759ebf50 con 0x55c675a8aa90 2024-03-24T00:43:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.263+0000 7f1ecf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 0x55c675a8a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.263+0000 7f1ecf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 0x55c675a8a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55788/0 (socket says 172.21.15.123:55788) 2024-03-24T00:43:42.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.263+0000 7f1ecf400640 1 -- 172.21.15.123:0/3513263920 learned_addr learned my addr 172.21.15.123:0/3513263920 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.264+0000 7f1ed4a00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 0x7f1eb40a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.264+0000 7f1ecea00640 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 0x55c675a8f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.264+0000 7f1ed4a00640 1 -- 172.21.15.123:0/4143560397 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1ec00cdc70 con 0x7f1ed010b2d0 2024-03-24T00:43:42.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.265+0000 7f1ecea00640 1 -- 172.21.15.123:0/3513263920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8fc00 msgr2=0x55c675a92090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.265+0000 7f1ecea00640 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8fc00 0x55c675a92090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.265+0000 7f1ecea00640 1 -- 172.21.15.123:0/3513263920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 msgr2=0x55c675a8a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.265+0000 7f1ecea00640 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 0x55c675a8a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.265+0000 7f1ecea00640 1 -- 172.21.15.123:0/3513263920 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6757cc7e0 con 0x55c675a8aa90 2024-03-24T00:43:42.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.266+0000 7f1ecf400640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 0x7f1eb40a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.266+0000 7f1ecea00640 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 0x55c675a8f590 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800f790 tx=0x7f1ec8007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5898d1ad4f79c621 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.267+0000 7f1ebbe00640 1 -- 172.21.15.123:0/3513263920 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8004d20 con 0x55c675a8aa90 2024-03-24T00:43:42.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.267+0000 7f1ecf400640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 0x7f1eb40a4e70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55c675a8b2f0 tx=0x7f1ebc02bd50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.267+0000 7f1ebbe00640 1 -- 172.21.15.123:0/3513263920 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ec800de00 con 0x55c675a8aa90 2024-03-24T00:43:42.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.267+0000 7f1ebbe00640 1 -- 172.21.15.123:0/3513263920 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8004d20 con 0x55c675a8aa90 2024-03-24T00:43:42.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.268+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3513263920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 msgr2=0x55c675a8f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.268+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 0x55c675a8f590 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800f790 tx=0x7f1ec8007f90 comp rx=0 tx=0).stop 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3513263920 shutdown_connections 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8fc00 0x55c675a92090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8aa90 0x55c675a8f590 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3513263920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8a0a0 0x55c675a8a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3513263920 >> 172.21.15.123:0/3513263920 conn(0x55c675a85610 msgr2=0x55c675a85a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3513263920 shutdown_connections 2024-03-24T00:43:42.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.269+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3513263920 wait complete. 2024-03-24T00:43:42.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.270+0000 7f1ed8bfb740 1 -- start start 2024-03-24T00:43:42.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.270+0000 7f1ed8bfb740 1 Processor -- start 2024-03-24T00:43:42.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.271+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a941a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.271+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.271+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a9b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ecea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ecea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a941a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37156/0 (socket says 172.21.15.123:37156) 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c675a92bb0 con 0x55c675a8fc00 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6759abeb0 con 0x55c675a8aa90 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c675a92d30 con 0x55c675a8a0a0 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.272+0000 7f1ed8bfb740 10 client.0 _ll_register_callbacks cb 0x55c6759a0150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:42.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39252/0 (socket says 172.21.15.123:39252) 2024-03-24T00:43:42.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 -- 172.21.15.123:0/392866808 learned_addr learned my addr 172.21.15.123:0/392866808 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ecf400640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a9b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 -- 172.21.15.123:0/392866808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 msgr2=0x55c675a941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 -- 172.21.15.123:0/392866808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 msgr2=0x55c675a9b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a9b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.273+0000 7f1ece000640 1 -- 172.21.15.123:0/392866808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c67582cf70 con 0x55c675a8aa90 2024-03-24T00:43:42.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.274+0000 7f1eb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:42.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.274+0000 7f1ece000640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00a2830 tx=0x7f1ec009ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72c2883c37f414f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.275+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00a2d60 con 0x55c675a8aa90 2024-03-24T00:43:42.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.275+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1ec000cd50 con 0x55c675a8aa90 2024-03-24T00:43:42.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.275+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ec800de00 con 0x55c675a8aa90 2024-03-24T00:43:42.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.278+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ec00a2d60 con 0x55c675a8aa90 2024-03-24T00:43:42.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.278+0000 7f1ed8bfb740 10 client.26174 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:42.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.278+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c6756c4170 con 0x55c675a8aa90 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.278+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ec800de00 con 0x55c675a8aa90 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.278+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1ec0099a10 con 0x55c675a8aa90 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 10 client.26174 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 10 client.26174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 20 client.26174 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675a9ff30 0x55c675aa2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 4 client.26174 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6759fd3f0 con 0x55c675a9ff30 2024-03-24T00:43:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.279+0000 7f1ecea00640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675a9ff30 0x55c675aa2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.281+0000 7f1eb8c00640 10 client.26174 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.283+0000 7f1ecea00640 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675a9ff30 0x55c675aa2350 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c976708cd7d7cd40 server_cookie=1d70457575ff9450 in_seq=0 out_seq=0 2024-03-24T00:43:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.285+0000 7f1eb8c00640 1 -- 172.21.15.123:0/392866808 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6759fd3f0 con 0x55c675a9ff30 2024-03-24T00:43:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.285+0000 7f1eb8c00640 10 client.26174 handle_command_reply: tid=0 2024-03-24T00:43:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.285+0000 7f1ed8bfb740 1 client.26174 shutdown 2024-03-24T00:43:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.287+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 msgr2=0x55c675a946e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.287+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00a2830 tx=0x7f1ec009ad80 comp rx=0 tx=0).stop 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 shutdown_connections 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a9b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a946e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675a9ff30 0x55c675aa2350 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/392866808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 >> 172.21.15.123:0/392866808 conn(0x55c675a85610 msgr2=0x55c675a84830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.291+0000 7f1eb8c00640 0 client.26174 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.292+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 shutdown_connections 2024-03-24T00:43:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.292+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/392866808 wait complete. 2024-03-24T00:43:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.292+0000 7f1ed8bfb740 20 client.26174 trim_cache size 0 max 16384 2024-03-24T00:43:42.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.331+0000 7f1ed8bfb740 1 Processor -- start 2024-03-24T00:43:42.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.331+0000 7f1ed8bfb740 1 -- start start 2024-03-24T00:43:42.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a8bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675aa6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000c540 con 0x55c675a8fc00 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000eb80 con 0x55c675a8aa90 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.332+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000cba0 con 0x55c675a8a0a0 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.333+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.333+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39254/0 (socket says 172.21.15.123:39254) 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.333+0000 7f1ece000640 1 -- 172.21.15.123:0/3279988386 learned_addr learned my addr 172.21.15.123:0/3279988386 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.333+0000 7f1ecea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a8bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.333+0000 7f1ecf400640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675aa6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ece000640 1 -- 172.21.15.123:0/3279988386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 msgr2=0x55c675a8bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ece000640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a8bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ece000640 1 -- 172.21.15.123:0/3279988386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 msgr2=0x55c675aa6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ece000640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675aa6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ece000640 1 -- 172.21.15.123:0/3279988386 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c675a80c20 con 0x55c675a8aa90 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ecea00640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a8bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.334+0000 7f1ecf400640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675aa6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:42.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.335+0000 7f1ece000640 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0098870 tx=0x7f1ec000d3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c107205671c214e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.335+0000 7f1ebb400640 1 -- 172.21.15.123:0/3279988386 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00a2d60 con 0x55c675a8aa90 2024-03-24T00:43:42.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.336+0000 7f1ebb400640 1 -- 172.21.15.123:0/3279988386 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ec0099d30 con 0x55c675a8aa90 2024-03-24T00:43:42.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.336+0000 7f1ebb400640 1 -- 172.21.15.123:0/3279988386 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00a28d0 con 0x55c675a8aa90 2024-03-24T00:43:42.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.337+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3279988386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 msgr2=0x55c675a9b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.337+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0098870 tx=0x7f1ec000d3f0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3279988386 shutdown_connections 2024-03-24T00:43:42.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675aa6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a9b5d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/3279988386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a8bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3279988386 >> 172.21.15.123:0/3279988386 conn(0x55c675a85610 msgr2=0x55c675a8b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3279988386 shutdown_connections 2024-03-24T00:43:42.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.338+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/3279988386 wait complete. 2024-03-24T00:43:42.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.339+0000 7f1ed8bfb740 1 -- start start 2024-03-24T00:43:42.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.339+0000 7f1ed8bfb740 1 Processor -- start 2024-03-24T00:43:42.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a9b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a99fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000d200 con 0x55c675a8fc00 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000e520 con 0x55c675a8aa90 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000d860 con 0x55c675a8a0a0 2024-03-24T00:43:42.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ed8bfb740 10 client.0 _ll_register_callbacks cb 0x7f1ec009a1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ece000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39270/0 (socket says 172.21.15.123:39270) 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.340+0000 7f1ece000640 1 -- 172.21.15.123:0/2527750881 learned_addr learned my addr 172.21.15.123:0/2527750881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ecea00640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a9b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ece000640 1 -- 172.21.15.123:0/2527750881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 msgr2=0x55c675a9b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ece000640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a9b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ecf400640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a99fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ece000640 1 -- 172.21.15.123:0/2527750881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 msgr2=0x55c675a99fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ece000640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a99fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.341+0000 7f1ece000640 1 -- 172.21.15.123:0/2527750881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c675a9a760 con 0x55c675a8aa90 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.342+0000 7f1ece000640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c640 tx=0x7f1ec009a4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d5da20b472b739 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.342+0000 7f1eb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:42.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.342+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00ba560 con 0x55c675a8aa90 2024-03-24T00:43:42.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.342+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1ec00a28d0 con 0x55c675a8aa90 2024-03-24T00:43:42.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.342+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ec0099d30 con 0x55c675a8aa90 2024-03-24T00:43:42.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.343+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ec0097d90 con 0x55c675a8aa90 2024-03-24T00:43:42.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.343+0000 7f1ed8bfb740 10 client.26180 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:42.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.343+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c675afc7d0 con 0x55c675a8aa90 2024-03-24T00:43:42.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.344+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ec0099d30 con 0x55c675a8aa90 2024-03-24T00:43:42.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.344+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1ec0099a10 con 0x55c675a8aa90 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 10 client.26180 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 10 client.26180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 20 client.26180 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675afcea0 0x55c675aff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 4 client.26180 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c6759f7740 con 0x55c675afcea0 2024-03-24T00:43:42.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.345+0000 7f1ecea00640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675afcea0 0x55c675aff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.346+0000 7f1eb8c00640 10 client.26180 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.346+0000 7f1ecea00640 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675afcea0 0x55c675aff2c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca1016c6a48c422 server_cookie=a06e9b5df97c7347 in_seq=0 out_seq=0 2024-03-24T00:43:42.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.348+0000 7f1eb8c00640 1 -- 172.21.15.123:0/2527750881 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c6759f7740 con 0x55c675afcea0 2024-03-24T00:43:42.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.348+0000 7f1eb8c00640 10 client.26180 handle_command_reply: tid=0 2024-03-24T00:43:42.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.348+0000 7f1ed8bfb740 1 client.26180 shutdown 2024-03-24T00:43:42.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.348+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 msgr2=0x55c675a99a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.348+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c640 tx=0x7f1ec009a4c0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 shutdown_connections 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c675a8fc00 0x55c675a99fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c675a8aa90 0x55c675a99a90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c675afcea0 0x55c675aff2c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 --2- 172.21.15.123:0/2527750881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c675a8a0a0 0x55c675a9b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 >> 172.21.15.123:0/2527750881 conn(0x55c675a85610 msgr2=0x55c675a8bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.349+0000 7f1eb8c00640 0 client.26180 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.350+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 shutdown_connections 2024-03-24T00:43:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.351+0000 7f1ed8bfb740 1 -- 172.21.15.123:0/2527750881 wait complete. 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.351+0000 7f1ed8bfb740 20 client.26180 trim_cache size 0 max 16384 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "ce6a6820-8bb1-41e9-a3c7-394d137aedb6", 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:43:42.351 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.352+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 msgr2=0x7f1eb40a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.352+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 0x7f1eb40a4e70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55c675a8b2f0 tx=0x7f1ebc02bd50 comp rx=0 tx=0).stop 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.352+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 msgr2=0x7f1ed01a6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.352+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00737f0 tx=0x7f1ec0009bc0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 shutdown_connections 2024-03-24T00:43:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1eb40a29b0 0x7f1eb40a4e70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed010dd70 0x7f1ed01add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed010b2d0 0x7f1ed01a6cb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 --2- 172.21.15.123:0/4143560397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ed0108910 0x7f1ed01a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.353+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 >> 172.21.15.123:0/4143560397 conn(0x7f1ed006e500 msgr2=0x7f1ed0109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.354+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 shutdown_connections 2024-03-24T00:43:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.354+0000 7f1ed5e00640 1 -- 172.21.15.123:0/4143560397 wait complete. 2024-03-24T00:43:42.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:43:42.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:43:42.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 -- 172.21.15.123:0/764878097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4101580 msgr2=0x7fddf41019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 --2- 172.21.15.123:0/764878097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4101580 0x7fddf41019a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdde8009900 tx=0x7fdde802bc50 comp rx=0 tx=0).stop 2024-03-24T00:43:42.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 -- 172.21.15.123:0/764878097 shutdown_connections 2024-03-24T00:43:42.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 --2- 172.21.15.123:0/764878097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4103130 0x7fddf41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 --2- 172.21.15.123:0/764878097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4102770 0x7fddf4102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 --2- 172.21.15.123:0/764878097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4101580 0x7fddf41019a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 -- 172.21.15.123:0/764878097 >> 172.21.15.123:0/764878097 conn(0x7fddf40fcb10 msgr2=0x7fddf40fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 -- 172.21.15.123:0/764878097 shutdown_connections 2024-03-24T00:43:42.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.597+0000 7fddfa200640 1 -- 172.21.15.123:0/764878097 wait complete. 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.598+0000 7fddfa200640 1 Processor -- start 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.598+0000 7fddfa200640 1 -- start start 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4101580 0x7fddf4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4102770 0x7fddf4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf410d240 con 0x7fddf4101580 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf410d0c0 con 0x7fddf4103130 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddfa200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf410d3c0 con 0x7fddf4102770 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4101580 0x7fddf4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4102770 0x7fddf4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39294/0 (socket says 172.21.15.123:39294) 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 -- 172.21.15.123:0/885618465 learned_addr learned my addr 172.21.15.123:0/885618465 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 -- 172.21.15.123:0/885618465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4102770 msgr2=0x7fddf4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4102770 0x7fddf4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 -- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4101580 msgr2=0x7fddf4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4101580 0x7fddf4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.599+0000 7fddf9800640 1 -- 172.21.15.123:0/885618465 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4074860 con 0x7fddf4103130 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddf9800640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fddec00eb10 tx=0x7fddec00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddf1600640 1 -- 172.21.15.123:0/885618465 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec004b20 con 0x7fddf4103130 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddf1600640 1 -- 172.21.15.123:0/885618465 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddec004cc0 con 0x7fddf4103130 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf41a1970 con 0x7fddf4103130 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddf1600640 1 -- 172.21.15.123:0/885618465 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec010760 con 0x7fddf4103130 2024-03-24T00:43:42.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.600+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41a1ec0 con 0x7fddf4103130 2024-03-24T00:43:42.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.603+0000 7fddfe5b4740 1 Processor -- start 2024-03-24T00:43:42.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.603+0000 7fddf1600640 1 -- 172.21.15.123:0/885618465 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fddec005020 con 0x7fddf4103130 2024-03-24T00:43:42.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.603+0000 7fddfe5b4740 1 -- start start 2024-03-24T00:43:42.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 0x5606c3df4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3d51250 con 0x5606c3deca40 2024-03-24T00:43:42.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3cf9a30 con 0x5606c3df1bb0 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3d4d2f0 con 0x5606c3dec050 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 0x5606c3df4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55840/0 (socket says 172.21.15.123:55840) 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.604+0000 7fddf8e00640 1 -- 172.21.15.123:0/1025334784 learned_addr learned my addr 172.21.15.123:0/1025334784 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 -- 172.21.15.123:0/1025334784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 msgr2=0x5606c3dec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 -- 172.21.15.123:0/1025334784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 msgr2=0x5606c3df4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf1600640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 0x7fdddc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 0x5606c3df4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 -- 172.21.15.123:0/1025334784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c3ce4690 con 0x5606c3deca40 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf3e00640 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 0x5606c3df4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:42.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf1600640 1 -- 172.21.15.123:0/885618465 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fddec0cf1d0 con 0x7fddf4103130 2024-03-24T00:43:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf3e00640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 0x7fdddc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fddf8e00640 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdde802c260 tx=0x7fdde8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7759b60b6968b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fdde6000640 1 -- 172.21.15.123:0/1025334784 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8042070 con 0x5606c3deca40 2024-03-24T00:43:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fdde6000640 1 -- 172.21.15.123:0/1025334784 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde802cba0 con 0x5606c3deca40 2024-03-24T00:43:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.605+0000 7fdde6000640 1 -- 172.21.15.123:0/1025334784 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8004eb0 con 0x5606c3deca40 2024-03-24T00:43:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 -- 172.21.15.123:0/1025334784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 msgr2=0x5606c3df1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddf3e00640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 0x7fdddc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5606c3ded2a0 tx=0x7fdde00046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdde802c260 tx=0x7fdde8009560 comp rx=0 tx=0).stop 2024-03-24T00:43:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 -- 172.21.15.123:0/1025334784 shutdown_connections 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3df1bb0 0x5606c3df4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df1540 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/1025334784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.606+0000 7fddfe5b4740 1 -- 172.21.15.123:0/1025334784 >> 172.21.15.123:0/1025334784 conn(0x5606c3db4200 msgr2=0x5606c3de79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 -- 172.21.15.123:0/1025334784 shutdown_connections 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 -- 172.21.15.123:0/1025334784 wait complete. 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 -- start start 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 Processor -- start 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.607+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 0x5606c3dfd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3df4b00 con 0x5606c3deca40 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3d0ce80 con 0x5606c3dec050 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c3df4c80 con 0x5606c3df1bb0 2024-03-24T00:43:42.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddfe5b4740 10 client.0 _ll_register_callbacks cb 0x5606c3d1f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39324/0 (socket says 172.21.15.123:39324) 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 -- 172.21.15.123:0/523170593 learned_addr learned my addr 172.21.15.123:0/523170593 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf9800640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 0x5606c3dfd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf3e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 -- 172.21.15.123:0/523170593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 msgr2=0x5606c3dfd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 0x5606c3dfd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 -- 172.21.15.123:0/523170593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 msgr2=0x5606c3df6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 -- 172.21.15.123:0/523170593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606c3b2f7d0 con 0x5606c3dec050 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf9800640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 0x5606c3dfd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fddf8e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdde8005ab0 tx=0x7fdde8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835b24c79c2d0a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.608+0000 7fdddaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde804c070 con 0x5606c3dec050 2024-03-24T00:43:42.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdde80050c0 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdde802cba0 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdde8058050 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fddfe5b4740 10 client.26192 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5606c3a653f0 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde802cba0 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.609+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdde8047c90 con 0x5606c3dec050 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 10 client.26192 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 20 client.26192 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e01a60 0x5606c3e03e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 4 client.26192 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606c3d5e790 con 0x5606c3e01a60 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddf3e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e01a60 0x5606c3e03e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fddf3e00640 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e01a60 0x5606c3e03e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ca7af99776e426 server_cookie=ea12e2ef5399a852 in_seq=0 out_seq=0 2024-03-24T00:43:42.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.610+0000 7fdddaa00640 10 client.26192 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fdddaa00640 1 -- 172.21.15.123:0/523170593 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5606c3d5e790 con 0x5606c3e01a60 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fdddaa00640 10 client.26192 handle_command_reply: tid=0 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 client.26192 shutdown 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 msgr2=0x5606c3df5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdde8005ab0 tx=0x7fdde8005850 comp rx=0 tx=0).stop 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 shutdown_connections 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e01a60 0x5606c3e03e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3df1bb0 0x5606c3dfd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3deca40 0x5606c3df6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/523170593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3dec050 0x5606c3df5ff0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fdddaa00640 0 client.26192 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.611+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 >> 172.21.15.123:0/523170593 conn(0x5606c3db4200 msgr2=0x5606c3de6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.612+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 shutdown_connections 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.612+0000 7fddfe5b4740 1 -- 172.21.15.123:0/523170593 wait complete. 2024-03-24T00:43:42.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.612+0000 7fddfe5b4740 20 client.26192 trim_cache size 0 max 16384 2024-03-24T00:43:42.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.629+0000 7fddfe5b4740 1 Processor -- start 2024-03-24T00:43:42.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.629+0000 7fddfe5b4740 1 -- start start 2024-03-24T00:43:42.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3e08600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3e08b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a8b00 con 0x5606c3df1bb0 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41ab0c0 con 0x5606c3deca40 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a9100 con 0x5606c3dec050 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3e08600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3e08600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37206/0 (socket says 172.21.15.123:37206) 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf8e00640 1 -- 172.21.15.123:0/4194249588 learned_addr learned my addr 172.21.15.123:0/4194249588 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf9800640 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3e08b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 -- 172.21.15.123:0/4194249588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 msgr2=0x5606c3e08600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3e08600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 -- 172.21.15.123:0/4194249588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 msgr2=0x5606c3e08b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3e08b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf3e00640 1 -- 172.21.15.123:0/4194249588 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c3a250f0 con 0x5606c3deca40 2024-03-24T00:43:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.630+0000 7fddf9800640 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3e08b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:42.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.631+0000 7fddf3e00640 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfd530 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5606c3e04550 tx=0x7fdde0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=520426b25501e1d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.631+0000 7fdde5600640 1 -- 172.21.15.123:0/4194249588 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0010cc0 con 0x5606c3deca40 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.631+0000 7fdde5600640 1 -- 172.21.15.123:0/4194249588 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde0032070 con 0x5606c3deca40 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.631+0000 7fdde5600640 1 -- 172.21.15.123:0/4194249588 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0004a00 con 0x5606c3deca40 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- 172.21.15.123:0/4194249588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 msgr2=0x5606c3dfd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfd530 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5606c3e04550 tx=0x7fdde0016000 comp rx=0 tx=0).stop 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- 172.21.15.123:0/4194249588 shutdown_connections 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3e08b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfd530 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/4194249588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3e08600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- 172.21.15.123:0/4194249588 >> 172.21.15.123:0/4194249588 conn(0x5606c3db4200 msgr2=0x5606c3dedce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- 172.21.15.123:0/4194249588 shutdown_connections 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- 172.21.15.123:0/4194249588 wait complete. 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 -- start start 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.632+0000 7fddfe5b4740 1 Processor -- start 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3dfc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a9740 con 0x5606c3df1bb0 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41aaa60 con 0x5606c3deca40 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a9da0 con 0x5606c3dec050 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddfe5b4740 10 client.0 _ll_register_callbacks cb 0x7fdde00099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37218/0 (socket says 172.21.15.123:37218) 2024-03-24T00:43:42.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 -- 172.21.15.123:0/86563791 learned_addr learned my addr 172.21.15.123:0/86563791 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf3e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf9800640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3dfc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 -- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 msgr2=0x5606c3dfb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 -- 172.21.15.123:0/86563791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 msgr2=0x5606c3dfc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3dfc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.633+0000 7fddf8e00640 1 -- 172.21.15.123:0/86563791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606c3dfc980 con 0x5606c3dec050 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fddf8e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdde802c7d0 tx=0x7fdde8005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52c874de01c0bdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fdddaa00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde804c070 con 0x5606c3dec050 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdde802cba0 con 0x5606c3dec050 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdde0032070 con 0x5606c3dec050 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdde8042040 con 0x5606c3dec050 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fddfe5b4740 10 client.26212 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5606c3e5d410 con 0x5606c3dec050 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.634+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde0032070 con 0x5606c3dec050 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdde80050c0 con 0x5606c3dec050 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 10 client.26212 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 20 client.26212 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e5dc80 0x5606c3e600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 4 client.26212 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5606c3d58ae0 con 0x5606c3e5dc80 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddf3e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e5dc80 0x5606c3e600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fddf3e00640 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e5dc80 0x5606c3e600a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94643ed7b1251632 server_cookie=3824348e415cd994 in_seq=0 out_seq=0 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.635+0000 7fdddaa00640 10 client.26212 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fdddaa00640 1 -- 172.21.15.123:0/86563791 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x5606c3d58ae0 con 0x5606c3e5dc80 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fdddaa00640 10 client.26212 handle_command_reply: tid=0 2024-03-24T00:43:42.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 client.26212 shutdown 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 msgr2=0x5606c3dfaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdde802c7d0 tx=0x7fdde8005b50 comp rx=0 tx=0).stop 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 shutdown_connections 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5606c3df1bb0 0x5606c3dfc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c3deca40 0x5606c3dfb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5606c3e5dc80 0x5606c3e600a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 --2- 172.21.15.123:0/86563791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606c3dec050 0x5606c3dfaeb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 >> 172.21.15.123:0/86563791 conn(0x5606c3db4200 msgr2=0x5606c3de6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fdddaa00640 0 client.26212 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 shutdown_connections 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 1 -- 172.21.15.123:0/86563791 wait complete. 2024-03-24T00:43:42.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.636+0000 7fddfe5b4740 20 client.26212 trim_cache size 0 max 16384 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (62 inodes in the stack)", 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "ce6a6820-8bb1-41e9-a3c7-394d137aedb6": { 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "ce6a6820-8bb1-41e9-a3c7-394d137aedb6", 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:43:42.641 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 msgr2=0x7fdddc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 0x7fdddc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5606c3ded2a0 tx=0x7fdde00046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 msgr2=0x7fddf4073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fddec00eb10 tx=0x7fddec00c2d0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 shutdown_connections 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf4103130 0x7fddf4073fd0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddf4102770 0x7fddf4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdddc0a2800 0x7fdddc0a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 --2- 172.21.15.123:0/885618465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddf4101580 0x7fddf4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 >> 172.21.15.123:0/885618465 conn(0x7fddf40fcb10 msgr2=0x7fddf4105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 shutdown_connections 2024-03-24T00:43:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:43:42.637+0000 7fddfa200640 1 -- 172.21.15.123:0/885618465 wait complete. 2024-03-24T00:43:42.651 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:42 smithi123 ceph-mon[31487]: pgmap v559: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 38 MiB/s wr, 4.31k op/s 2024-03-24T00:43:42.655 INFO:tasks.cephfs.filesystem:scrub status for tag:ce6a6820-8bb1-41e9-a3c7-394d137aedb6 - {'path': '/', 'tag': 'ce6a6820-8bb1-41e9-a3c7-394d137aedb6', 'options': 'recursive,force'} 2024-03-24T00:43:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:42 smithi177 ceph-mon[36034]: pgmap v559: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 38 MiB/s wr, 4.31k op/s 2024-03-24T00:43:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:42 smithi204 ceph-mon[36345]: pgmap v559: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 38 MiB/s wr, 4.31k op/s 2024-03-24T00:43:43.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:43 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:43:43.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:43 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:43.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:43 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:43:43.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:43 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:43:43.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:43 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:43.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:43 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:43:43.860 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:43 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:43:43.860 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:43 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:43.860 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:43 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:43:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:44 smithi177 ceph-mon[36034]: pgmap v560: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 37 MiB/s wr, 4.00k op/s 2024-03-24T00:43:44.758 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:44 smithi123 ceph-mon[31487]: pgmap v560: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 37 MiB/s wr, 4.00k op/s 2024-03-24T00:43:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:44 smithi204 ceph-mon[36345]: pgmap v560: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 37 MiB/s wr, 4.00k op/s 2024-03-24T00:43:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:46 smithi177 ceph-mon[36034]: pgmap v561: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 39 MiB/s wr, 4.26k op/s 2024-03-24T00:43:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:46 smithi204 ceph-mon[36345]: pgmap v561: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 39 MiB/s wr, 4.26k op/s 2024-03-24T00:43:46.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:46 smithi123 ceph-mon[31487]: pgmap v561: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 39 MiB/s wr, 4.26k op/s 2024-03-24T00:43:47.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:47 smithi177 ceph-mon[36034]: pgmap v562: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 34 MiB/s wr, 3.67k op/s 2024-03-24T00:43:47.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:47 smithi204 ceph-mon[36345]: pgmap v562: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 34 MiB/s wr, 3.67k op/s 2024-03-24T00:43:47.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:47 smithi123 ceph-mon[31487]: pgmap v562: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 34 MiB/s wr, 3.67k op/s 2024-03-24T00:43:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:48 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:48 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:48 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:43:49.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:49 smithi177 ceph-mon[36034]: pgmap v563: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 37 MiB/s wr, 4.02k op/s 2024-03-24T00:43:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:49 smithi204 ceph-mon[36345]: pgmap v563: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 37 MiB/s wr, 4.02k op/s 2024-03-24T00:43:49.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:49 smithi123 ceph-mon[31487]: pgmap v563: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 37 MiB/s wr, 4.02k op/s 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 -- 172.21.15.123:0/283290425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 msgr2=0x7f9c88104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 --2- 172.21.15.123:0/283290425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c88104f70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009860 tx=0x7f9c7402bb70 comp rx=0 tx=0).stop 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 -- 172.21.15.123:0/283290425 shutdown_connections 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 --2- 172.21.15.123:0/283290425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c881054b0 0x7f9c8810bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 --2- 172.21.15.123:0/283290425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c88104f70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 --2- 172.21.15.123:0/283290425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c88103900 0x7f9c88103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 -- 172.21.15.123:0/283290425 >> 172.21.15.123:0/283290425 conn(0x7f9c880fee90 msgr2=0x7f9c881012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 -- 172.21.15.123:0/283290425 shutdown_connections 2024-03-24T00:43:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.385+0000 7f9c8f000640 1 -- 172.21.15.123:0/283290425 wait complete. 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 Processor -- start 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 -- start start 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 0x7f9c88199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c8819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8810f580 con 0x7f9c88104af0 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8810f400 con 0x7f9c881054b0 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.386+0000 7f9c8f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8810f700 con 0x7f9c88103900 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 0x7f9c88199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54776/0 (socket says 172.21.15.123:54776) 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 -- 172.21.15.123:0/2889830640 learned_addr learned my addr 172.21.15.123:0/2889830640 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 0x7f9c88199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:47160/0 (socket says 172.21.15.123:47160) 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c87400640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c8819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 -- 172.21.15.123:0/2889830640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 msgr2=0x7f9c88199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 0x7f9c88199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 -- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 msgr2=0x7f9c8819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c8819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 -- 172.21.15.123:0/2889830640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c881a1720 con 0x7f9c881054b0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.387+0000 7f9c8d000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000eb10 tx=0x7f9c8000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.388+0000 7f9c84c00640 1 -- 172.21.15.123:0/2889830640 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80004b20 con 0x7f9c881054b0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.388+0000 7f9c84c00640 1 -- 172.21.15.123:0/2889830640 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c80004cc0 con 0x7f9c881054b0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.388+0000 7f9c84c00640 1 -- 172.21.15.123:0/2889830640 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80010760 con 0x7f9c881054b0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.388+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c881a1a10 con 0x7f9c881054b0 2024-03-24T00:43:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.388+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c881a1f80 con 0x7f9c881054b0 2024-03-24T00:43:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.391+0000 7f9c84c00640 1 -- 172.21.15.123:0/2889830640 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9c80005020 con 0x7f9c881054b0 2024-03-24T00:43:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.391+0000 7f9c91c16740 1 Processor -- start 2024-03-24T00:43:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 -- start start 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f911e0 0x55f439f91600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439f991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439f543b0 con 0x55f439f96d40 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439ef6830 con 0x55f439f91bd0 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c91c16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439e9f5d0 con 0x55f439f911e0 2024-03-24T00:43:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54780/0 (socket says 172.21.15.123:54780) 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c87e00640 1 -- 172.21.15.123:0/3976221319 learned_addr learned my addr 172.21.15.123:0/3976221319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.392+0000 7f9c87400640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f911e0 0x55f439f91600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87400640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439f991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 -- 172.21.15.123:0/3976221319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f911e0 msgr2=0x55f439f91600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f911e0 0x55f439f91600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 -- 172.21.15.123:0/3976221319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 msgr2=0x55f439f991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439f991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 -- 172.21.15.123:0/3976221319 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f439eb72c0 con 0x55f439f91bd0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87400640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439f991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87e00640 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009c80 tx=0x7f9c78011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0696a36efd6a1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c84c00640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 0x7f9c6c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c7d600640 1 -- 172.21.15.123:0/3976221319 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78005220 con 0x55f439f91bd0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c84c00640 1 -- 172.21.15.123:0/2889830640 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c800cf100 con 0x7f9c881054b0 2024-03-24T00:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c7d600640 1 -- 172.21.15.123:0/3976221319 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c78009e30 con 0x55f439f91bd0 2024-03-24T00:43:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c7d600640 1 -- 172.21.15.123:0/3976221319 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78019810 con 0x55f439f91bd0 2024-03-24T00:43:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.393+0000 7f9c87400640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 0x7f9c6c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 -- 172.21.15.123:0/3976221319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 msgr2=0x55f439f966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009c80 tx=0x7f9c78011810 comp rx=0 tx=0).stop 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c87400640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 0x7f9c6c0a4de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402c180 tx=0x7f9c74034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 -- 172.21.15.123:0/3976221319 shutdown_connections 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439f991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f966d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 --2- 172.21.15.123:0/3976221319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f911e0 0x55f439f91600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 -- 172.21.15.123:0/3976221319 >> 172.21.15.123:0/3976221319 conn(0x55f439f8c750 msgr2=0x55f439f8cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 -- 172.21.15.123:0/3976221319 shutdown_connections 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.394+0000 7f9c91c16740 1 -- 172.21.15.123:0/3976221319 wait complete. 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 -- start start 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 Processor -- start 2024-03-24T00:43:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f91bd0 0x55f439f9b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439fa2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439eb2a20 con 0x55f439f91bd0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439ef28d0 con 0x55f439f911e0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f439f99cf0 con 0x55f439f96d40 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.395+0000 7f9c91c16740 10 client.0 _ll_register_callbacks cb 0x55f439eef1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54796/0 (socket says 172.21.15.123:54796) 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 -- 172.21.15.123:0/1754306058 learned_addr learned my addr 172.21.15.123:0/1754306058 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87400640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f91bd0 0x55f439f9b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c8d000640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439fa2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 -- 172.21.15.123:0/1754306058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 msgr2=0x55f439fa2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439fa2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 -- 172.21.15.123:0/1754306058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f91bd0 msgr2=0x55f439f9b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f91bd0 0x55f439f9b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.396+0000 7f9c87e00640 1 -- 172.21.15.123:0/1754306058 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f439c0b210 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c87e00640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9c7801b040 tx=0x7f9c78019c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e1735a9b34596f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c72000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78029070 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c78023840 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c78009e30 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c78035050 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c91c16740 10 client.26210 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f439cd7620 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.397+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c78009e30 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9c7800e7c0 con 0x55f439f911e0 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 10 client.26210 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 10 client.26210 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 20 client.26210 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f439fa6e10 0x55f439fa9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 4 client.26210 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f439f03d70 con 0x55f439fa6e10 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.398+0000 7f9c87400640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f439fa6e10 0x55f439fa9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.399+0000 7f9c87400640 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f439fa6e10 0x55f439fa9200 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6cafc0115a51a29 server_cookie=ef55cf9d180f2204 in_seq=0 out_seq=0 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.399+0000 7f9c72000640 10 client.26210 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.400+0000 7f9c72000640 1 -- 172.21.15.123:0/1754306058 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f439f03d70 con 0x55f439fa6e10 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.400+0000 7f9c72000640 10 client.26210 handle_command_reply: tid=0 2024-03-24T00:43:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.400+0000 7f9c91c16740 1 client.26210 shutdown 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 msgr2=0x55f439f9b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9c7801b040 tx=0x7f9c78019c80 comp rx=0 tx=0).stop 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 shutdown_connections 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f439fa6e10 0x55f439fa9200 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439fa2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f91bd0 0x55f439f9b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1754306058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9b3f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 >> 172.21.15.123:0/1754306058 conn(0x55f439f8c750 msgr2=0x55f439f8b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c72000640 0 client.26210 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 shutdown_connections 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 1 -- 172.21.15.123:0/1754306058 wait complete. 2024-03-24T00:43:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.401+0000 7f9c91c16740 20 client.26210 trim_cache size 0 max 16384 2024-03-24T00:43:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 Processor -- start 2024-03-24T00:43:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 -- start start 2024-03-24T00:43:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f911e0 0x55f439f8b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439f8bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881a8cd0 con 0x55f439f911e0 2024-03-24T00:43:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881ab290 con 0x55f439f91bd0 2024-03-24T00:43:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.418+0000 7f9c91c16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881a92d0 con 0x55f439f96d40 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c8d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439f8bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54810/0 (socket says 172.21.15.123:54810) 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f911e0 0x55f439f8b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 -- 172.21.15.123:0/1969183848 learned_addr learned my addr 172.21.15.123:0/1969183848 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c8d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439f8bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:47188/0 (socket says 172.21.15.123:47188) 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 -- 172.21.15.123:0/1969183848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 msgr2=0x55f439f8bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439f8bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 -- 172.21.15.123:0/1969183848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f911e0 msgr2=0x55f439f8b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f911e0 0x55f439f8b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 -- 172.21.15.123:0/1969183848 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f439f99790 con 0x55f439f91bd0 2024-03-24T00:43:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.419+0000 7f9c87400640 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402e010 tx=0x7f9c74041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c7ddf8645c5e643 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.420+0000 7f9c7cc00640 1 -- 172.21.15.123:0/1969183848 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74039040 con 0x55f439f91bd0 2024-03-24T00:43:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.420+0000 7f9c7cc00640 1 -- 172.21.15.123:0/1969183848 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c74052cc0 con 0x55f439f91bd0 2024-03-24T00:43:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.420+0000 7f9c7cc00640 1 -- 172.21.15.123:0/1969183848 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7402ca70 con 0x55f439f91bd0 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- 172.21.15.123:0/1969183848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 msgr2=0x55f439f8b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402e010 tx=0x7f9c74041000 comp rx=0 tx=0).stop 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- 172.21.15.123:0/1969183848 shutdown_connections 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f96d40 0x55f439f8bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f91bd0 0x55f439f8b600 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 --2- 172.21.15.123:0/1969183848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f911e0 0x55f439f8b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- 172.21.15.123:0/1969183848 >> 172.21.15.123:0/1969183848 conn(0x55f439f8c750 msgr2=0x55f439f8cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- 172.21.15.123:0/1969183848 shutdown_connections 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- 172.21.15.123:0/1969183848 wait complete. 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 -- start start 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.421+0000 7f9c91c16740 1 Processor -- start 2024-03-24T00:43:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 0x55f439f9fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439fa0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881a9910 con 0x55f439f96d40 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881aac30 con 0x55f439f911e0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881a9f70 con 0x55f439f91bd0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c91c16740 10 client.0 _ll_register_callbacks cb 0x7f9c78009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 0x55f439f9fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54814/0 (socket says 172.21.15.123:54814) 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87e00640 1 -- 172.21.15.123:0/801531340 learned_addr learned my addr 172.21.15.123:0/801531340 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c8d000640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439fa0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 -- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 msgr2=0x55f439f9f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 -- 172.21.15.123:0/801531340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 msgr2=0x55f439fa0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439fa0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87400640 1 -- 172.21.15.123:0/801531340 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f439fa1230 con 0x55f439f91bd0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c87e00640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.422+0000 7f9c8d000640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439fa0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c87400640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 0x55f439f9fc30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9c740349e0 tx=0x7f9c74034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a537dd0d59fdcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c72000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74060070 con 0x55f439f91bd0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c74052cc0 con 0x55f439f91bd0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c74052160 con 0x55f439f91bd0 2024-03-24T00:43:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7406c050 con 0x55f439f91bd0 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c91c16740 10 client.26230 fetch_fsmap learned FSMap version 19 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.423+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f43a0026b0 con 0x55f439f91bd0 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c74052cc0 con 0x55f439f91bd0 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9c7403f040 con 0x55f439f91bd0 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 10 client.26230 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 10 client.26230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 20 client.26230 populate_metadata read hostname 'smithi123' 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f43a002f40 0x55f43a005330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 4 client.26230 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f439efe0c0 con 0x55f43a002f40 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.424+0000 7f9c87e00640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f43a002f40 0x55f43a005330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.425+0000 7f9c72000640 10 client.26230 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:43:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:43:51.425+0000 7f9c87e00640 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f43a002f40 0x55f43a005330 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dae2318b04d56eb server_cookie=993f7a54c7f5444f in_seq=0 out_seq=0 2024-03-24T00:43:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:52 smithi123 ceph-mon[31487]: pgmap v564: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 40 MiB/s wr, 4.27k op/s 2024-03-24T00:43:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:52 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:43:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:52 smithi177 ceph-mon[36034]: pgmap v564: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 40 MiB/s wr, 4.27k op/s 2024-03-24T00:43:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:52 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:43:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:52 smithi204 ceph-mon[36345]: pgmap v564: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 40 MiB/s wr, 4.27k op/s 2024-03-24T00:43:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:52 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:43:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:54 smithi123 ceph-mon[31487]: pgmap v565: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 38 MiB/s wr, 4.14k op/s 2024-03-24T00:43:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:54 smithi177 ceph-mon[36034]: pgmap v565: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 38 MiB/s wr, 4.14k op/s 2024-03-24T00:43:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:54 smithi204 ceph-mon[36345]: pgmap v565: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 38 MiB/s wr, 4.14k op/s 2024-03-24T00:43:56.623 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:56 smithi204 ceph-mon[36345]: pgmap v566: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 34 MiB/s wr, 3.68k op/s 2024-03-24T00:43:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:56 smithi177 ceph-mon[36034]: pgmap v566: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 34 MiB/s wr, 3.68k op/s 2024-03-24T00:43:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:56 smithi123 ceph-mon[31487]: pgmap v566: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 34 MiB/s wr, 3.68k op/s 2024-03-24T00:43:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:43:58 smithi177 ceph-mon[36034]: pgmap v567: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 29 MiB/s wr, 3.04k op/s 2024-03-24T00:43:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:43:58 smithi204 ceph-mon[36345]: pgmap v567: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 29 MiB/s wr, 3.04k op/s 2024-03-24T00:43:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:43:58 smithi123 ceph-mon[31487]: pgmap v567: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 29 MiB/s wr, 3.04k op/s 2024-03-24T00:44:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:00 smithi177 ceph-mon[36034]: pgmap v568: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 35 MiB/s wr, 3.58k op/s 2024-03-24T00:44:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:00 smithi204 ceph-mon[36345]: pgmap v568: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 35 MiB/s wr, 3.58k op/s 2024-03-24T00:44:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:00 smithi123 ceph-mon[31487]: pgmap v568: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 35 MiB/s wr, 3.58k op/s 2024-03-24T00:44:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:02 smithi177 ceph-mon[36034]: pgmap v569: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 39 MiB/s wr, 3.98k op/s 2024-03-24T00:44:02.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:02 smithi204 ceph-mon[36345]: pgmap v569: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 39 MiB/s wr, 3.98k op/s 2024-03-24T00:44:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:02 smithi123 ceph-mon[31487]: pgmap v569: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 39 MiB/s wr, 3.98k op/s 2024-03-24T00:44:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:04 smithi177 ceph-mon[36034]: pgmap v570: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 39 MiB/s wr, 3.73k op/s 2024-03-24T00:44:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:04 smithi204 ceph-mon[36345]: pgmap v570: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 39 MiB/s wr, 3.73k op/s 2024-03-24T00:44:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:04 smithi123 ceph-mon[31487]: pgmap v570: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 39 MiB/s wr, 3.73k op/s 2024-03-24T00:44:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:06 smithi177 ceph-mon[36034]: pgmap v571: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 40 MiB/s wr, 3.70k op/s 2024-03-24T00:44:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:06 smithi204 ceph-mon[36345]: pgmap v571: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 40 MiB/s wr, 3.70k op/s 2024-03-24T00:44:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:06 smithi123 ceph-mon[31487]: pgmap v571: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 40 MiB/s wr, 3.70k op/s 2024-03-24T00:44:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:08 smithi177 ceph-mon[36034]: pgmap v572: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:44:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:08 smithi204 ceph-mon[36345]: pgmap v572: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:44:08.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:08 smithi123 ceph-mon[31487]: pgmap v572: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:44:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:10 smithi177 ceph-mon[36034]: pgmap v573: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.71k op/s 2024-03-24T00:44:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.801+0000 7f9c72000640 1 -- 172.21.15.123:0/801531340 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f439efe0c0 con 0x55f43a002f40 2024-03-24T00:44:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.801+0000 7f9c72000640 10 client.26230 handle_command_reply: tid=0 2024-03-24T00:44:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.801+0000 7f9c91c16740 1 client.26230 shutdown 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.802+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 msgr2=0x55f439f9fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.802+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 0x55f439f9fc30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9c740349e0 tx=0x7f9c74034440 comp rx=0 tx=0).stop 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 shutdown_connections 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f439f96d40 0x55f439fa0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f43a002f40 0x55f43a005330 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f439f91bd0 0x55f439f9fc30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 --2- 172.21.15.123:0/801531340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f439f911e0 0x55f439f9f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 >> 172.21.15.123:0/801531340 conn(0x55f439f8c750 msgr2=0x55f439f8f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.803+0000 7f9c72000640 0 client.26230 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.805+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 shutdown_connections 2024-03-24T00:44:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.805+0000 7f9c91c16740 1 -- 172.21.15.123:0/801531340 wait complete. 2024-03-24T00:44:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.806+0000 7f9c91c16740 20 client.26230 trim_cache size 0 max 16384 2024-03-24T00:44:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:44:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:44:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:44:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:44:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.807+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 msgr2=0x7f9c6c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.807+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 0x7f9c6c0a4de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402c180 tx=0x7f9c74034ad0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.807+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 msgr2=0x7f9c881a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.807+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000eb10 tx=0x7f9c8000c2d0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 shutdown_connections 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c881054b0 0x7f9c881a10b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9c6c0a2920 0x7f9c6c0a4de0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c88104af0 0x7f9c8819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 --2- 172.21.15.123:0/2889830640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c88103900 0x7f9c88199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 >> 172.21.15.123:0/2889830640 conn(0x7f9c880fee90 msgr2=0x7f9c88107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 shutdown_connections 2024-03-24T00:44:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:10.808+0000 7f9c8f000640 1 -- 172.21.15.123:0/2889830640 wait complete. 2024-03-24T00:44:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:10 smithi204 ceph-mon[36345]: pgmap v573: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.71k op/s 2024-03-24T00:44:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:10 smithi123 ceph-mon[31487]: pgmap v573: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.71k op/s 2024-03-24T00:44:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:11.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:11 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:11 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:11.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:11.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:11 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:44:12.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:44:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-24T00:44:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: pgmap v574: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.65k op/s 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:44:12.736 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-24T00:44:12.736 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-24T00:44:12.736 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: pgmap v574: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.65k op/s 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:12.737 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: pgmap v574: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 42 MiB/s wr, 3.65k op/s 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:44:12.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:12.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:12.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 -- 172.21.15.123:0/3348548794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 msgr2=0x7fcbc81043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 --2- 172.21.15.123:0/3348548794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc81043f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009900 tx=0x7fcbbc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 -- 172.21.15.123:0/3348548794 shutdown_connections 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 --2- 172.21.15.123:0/3348548794 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 --2- 172.21.15.123:0/3348548794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 0x7fcbc8106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 --2- 172.21.15.123:0/3348548794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc81043f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 -- 172.21.15.123:0/3348548794 >> 172.21.15.123:0/3348548794 conn(0x7fcbc80fba10 msgr2=0x7fcbc80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 -- 172.21.15.123:0/3348548794 shutdown_connections 2024-03-24T00:44:12.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.872+0000 7fcbce200640 1 -- 172.21.15.123:0/3348548794 wait complete. 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 Processor -- start 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 -- start start 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 0x7fcbc80fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc810d260 con 0x7fcbc8104930 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc810d0e0 con 0x7fcbc8102000 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbce200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc810d3e0 con 0x7fcbc8107260 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.873+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47914/0 (socket says 172.21.15.123:47914) 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 -- 172.21.15.123:0/1988621574 learned_addr learned my addr 172.21.15.123:0/1988621574 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcd800640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 0x7fcbc80fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 -- 172.21.15.123:0/1988621574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 msgr2=0x7fcbc81004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 -- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 msgr2=0x7fcbc80fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 0x7fcbc80fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 -- 172.21.15.123:0/1988621574 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8100d80 con 0x7fcbc8102000 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcd800640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.874+0000 7fcbcce00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000c00 tx=0x7fcbbc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.875+0000 7fcbc5600640 1 -- 172.21.15.123:0/1988621574 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc043020 con 0x7fcbc8102000 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.875+0000 7fcbc5600640 1 -- 172.21.15.123:0/1988621574 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc0054e0 con 0x7fcbc8102000 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.875+0000 7fcbc5600640 1 -- 172.21.15.123:0/1988621574 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc003c40 con 0x7fcbc8102000 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.875+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc81a19d0 con 0x7fcbc8102000 2024-03-24T00:44:12.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.875+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc81a1ed0 con 0x7fcbc8102000 2024-03-24T00:44:12.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.878+0000 7fcbc5600640 1 -- 172.21.15.123:0/1988621574 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcbbc046020 con 0x7fcbc8102000 2024-03-24T00:44:12.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.878+0000 7fcbd25d2740 1 Processor -- start 2024-03-24T00:44:12.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 -- start start 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fda40 0x5630a8702540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a8705040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a8662250 con 0x5630a8702bb0 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a860aa30 con 0x5630a86fda40 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a865e2f0 con 0x5630a86fd050 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fda40 0x5630a8702540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58454/0 (socket says 172.21.15.123:58454) 2024-03-24T00:44:12.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbcd800640 1 -- 172.21.15.123:0/3047690307 learned_addr learned my addr 172.21.15.123:0/3047690307 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.879+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a8705040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 -- 172.21.15.123:0/3047690307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fda40 msgr2=0x5630a8702540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fda40 0x5630a8702540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 -- 172.21.15.123:0/3047690307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 msgr2=0x5630a8705040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a8705040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 -- 172.21.15.123:0/3047690307 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a85f5690 con 0x5630a86fd050 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00117e0 tx=0x7fcbc0011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404c257c0d329265 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbba000640 1 -- 172.21.15.123:0/3047690307 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011310 con 0x5630a86fd050 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbba000640 1 -- 172.21.15.123:0/3047690307 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0003e50 con 0x5630a86fd050 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbba000640 1 -- 172.21.15.123:0/3047690307 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0004550 con 0x5630a86fd050 2024-03-24T00:44:12.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.880+0000 7fcbc5600640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbc5600640 1 -- 172.21.15.123:0/1988621574 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcbbc0f0a70 con 0x7fcbc8102000 2024-03-24T00:44:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbcce00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3047690307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 msgr2=0x5630a86fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00117e0 tx=0x7fcbc0011ca0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3047690307 shutdown_connections 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a8705040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fda40 0x5630a8702540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3047690307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fd050 0x5630a86fd470 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3047690307 >> 172.21.15.123:0/3047690307 conn(0x5630a86c5200 msgr2=0x5630a86f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3047690307 shutdown_connections 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3047690307 wait complete. 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.881+0000 7fcbcce00640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 0x7fcbb00a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0b9d90 tx=0x7fcbbc0b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 -- start start 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 Processor -- start 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 0x5630a8707040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 0x5630a870e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.882+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a8705b60 con 0x5630a86fd050 2024-03-24T00:44:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a861de80 con 0x5630a8702bb0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a8705ce0 con 0x5630a86fda40 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbd25d2740 10 client.0 _ll_register_callbacks cb 0x5630a8630130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 0x5630a8707040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58460/0 (socket says 172.21.15.123:58460) 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 0x5630a8707040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40240/0 (socket says 172.21.15.123:40240) 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 -- 172.21.15.123:0/3898459167 learned_addr learned my addr 172.21.15.123:0/3898459167 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 0x5630a870e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 -- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 msgr2=0x5630a870e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 0x5630a870e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 -- 172.21.15.123:0/3898459167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 msgr2=0x5630a8707040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 0x5630a8707040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbc7e00640 1 -- 172.21.15.123:0/3898459167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630a84407d0 con 0x5630a86fda40 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.883+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 0x5630a870e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4012700 tx=0x7fcbb4012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=777740d40ea6fc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbaea00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400f8c0 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcbb400fa60 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc0003e50 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb400fc60 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbd25d2740 10 client.26242 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5630a83763f0 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.884+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0003e50 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcbb401c420 con 0x5630a86fda40 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 10 client.26242 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 10 client.26242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 20 client.26242 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a8712e00 0x5630a8715260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 4 client.26242 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.885+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630a866f790 con 0x5630a8712e00 2024-03-24T00:44:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.886+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a8712e00 0x5630a8715260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.886+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a8712e00 0x5630a8715260 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a72b01735450534 server_cookie=26112e78cbbfc2ad in_seq=0 out_seq=0 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.886+0000 7fcbaea00640 10 client.26242 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.887+0000 7fcbaea00640 1 -- 172.21.15.123:0/3898459167 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5630a866f790 con 0x5630a8712e00 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.887+0000 7fcbaea00640 10 client.26242 handle_command_reply: tid=0 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.887+0000 7fcbd25d2740 1 client.26242 shutdown 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 msgr2=0x5630a8707580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4012700 tx=0x7fcbb4012bc0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 shutdown_connections 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a8702bb0 0x5630a870e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a8712e00 0x5630a8715260 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a8707580 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3898459167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a86fd050 0x5630a8707040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbaea00640 0 client.26242 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 >> 172.21.15.123:0/3898459167 conn(0x5630a86c5200 msgr2=0x5630a86f7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 shutdown_connections 2024-03-24T00:44:12.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3898459167 wait complete. 2024-03-24T00:44:12.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.888+0000 7fcbd25d2740 20 client.26242 trim_cache size 0 max 16384 2024-03-24T00:44:12.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 Processor -- start 2024-03-24T00:44:12.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 -- start start 2024-03-24T00:44:12.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81a8b20 con 0x5630a8702bb0 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81ab0e0 con 0x5630a86fd050 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.908+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81a9120 con 0x5630a86fda40 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47954/0 (socket says 172.21.15.123:47954) 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58466/0 (socket says 172.21.15.123:58466) 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 -- 172.21.15.123:0/3508324779 learned_addr learned my addr 172.21.15.123:0/3508324779 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 -- 172.21.15.123:0/3508324779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 msgr2=0x5630a86f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 -- 172.21.15.123:0/3508324779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 msgr2=0x5630a870e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 -- 172.21.15.123:0/3508324779 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a83360f0 con 0x5630a86fd050 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:12.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbcd800640 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00117b0 tx=0x7fcbc000ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=787fa63a26f7c042 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.909+0000 7fcbb9600640 1 -- 172.21.15.123:0/3508324779 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0023040 con 0x5630a86fd050 2024-03-24T00:44:12.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbb9600640 1 -- 172.21.15.123:0/3508324779 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc000fba0 con 0x5630a86fd050 2024-03-24T00:44:12.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbb9600640 1 -- 172.21.15.123:0/3508324779 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc001a6a0 con 0x5630a86fd050 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3508324779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 msgr2=0x5630a86f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00117b0 tx=0x7fcbc000ff50 comp rx=0 tx=0).stop 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3508324779 shutdown_connections 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a86f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/3508324779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a86f7a40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3508324779 >> 172.21.15.123:0/3508324779 conn(0x5630a86c5200 msgr2=0x5630a86f7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.910+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3508324779 shutdown_connections 2024-03-24T00:44:12.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 -- 172.21.15.123:0/3508324779 wait complete. 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 -- start start 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 Processor -- start 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a870c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.911+0000 7fcbd25d2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81a9760 con 0x5630a8702bb0 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81aaa80 con 0x5630a86fd050 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbd25d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81a9dc0 con 0x5630a86fda40 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbd25d2740 10 client.0 _ll_register_callbacks cb 0x7fcbb4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a870c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbcd800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58472/0 (socket says 172.21.15.123:58472) 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 -- 172.21.15.123:0/830978763 learned_addr learned my addr 172.21.15.123:0/830978763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:12.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 -- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 msgr2=0x5630a870c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a870c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 -- 172.21.15.123:0/830978763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 msgr2=0x5630a870d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 -- 172.21.15.123:0/830978763 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630a870de80 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbc7e00640 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54dff1b3216bd466 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.912+0000 7fcbaea00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402b030 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcbb4018040 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc000fba0 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb400f790 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbd25d2740 10 client.26248 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5630a876eeb0 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.913+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbc000fba0 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcbb4003890 con 0x5630a86fda40 2024-03-24T00:44:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 10 client.26248 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 10 client.26248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 20 client.26248 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a876f770 0x5630a8771b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 4 client.26248 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5630a8669ae0 con 0x5630a876f770 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbcd800640 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a876f770 0x5630a8771b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbcd800640 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a876f770 0x5630a8771b90 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3af30f716b2a4b13 server_cookie=f56809c92d18be9c in_seq=0 out_seq=0 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.914+0000 7fcbaea00640 10 client.26248 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbaea00640 1 -- 172.21.15.123:0/830978763 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5630a8669ae0 con 0x5630a876f770 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbaea00640 10 client.26248 handle_command_reply: tid=0 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 client.26248 shutdown 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 msgr2=0x5630a870c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4009f90 comp rx=0 tx=0).stop 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 shutdown_connections 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5630a8702bb0 0x5630a870d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5630a876f770 0x5630a8771b90 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5630a86fda40 0x5630a870c880 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 --2- 172.21.15.123:0/830978763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5630a86fd050 0x5630a870c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbaea00640 0 client.26248 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 >> 172.21.15.123:0/830978763 conn(0x5630a86c5200 msgr2=0x5630a86f7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 shutdown_connections 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 1 -- 172.21.15.123:0/830978763 wait complete. 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.915+0000 7fcbd25d2740 20 client.26248 trim_cache size 0 max 16384 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:44:12.920 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 msgr2=0x7fcbb00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 0x7fcbb00a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0b9d90 tx=0x7fcbbc0b9040 comp rx=0 tx=0).stop 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 msgr2=0x7fcbc8101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000c00 tx=0x7fcbbc02c650 comp rx=0 tx=0).stop 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 shutdown_connections 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc8107260 0x7fcbc81004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc8104930 0x7fcbc80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 --2- 172.21.15.123:0/1988621574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbc8102000 0x7fcbc8101970 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:12.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 >> 172.21.15.123:0/1988621574 conn(0x7fcbc80fba10 msgr2=0x7fcbc8102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:12.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 shutdown_connections 2024-03-24T00:44:12.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:12.916+0000 7fcbce200640 1 -- 172.21.15.123:0/1988621574 wait complete. 2024-03-24T00:44:12.936 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:44:12.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:13.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.205+0000 7f438fe00640 1 -- 172.21.15.123:0/3162063689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43901050c0 msgr2=0x7f4390105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.205+0000 7f438fe00640 1 --2- 172.21.15.123:0/3162063689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43901050c0 0x7f4390105540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4388009be0 tx=0x7f438802bef0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.205+0000 7f438fe00640 1 -- 172.21.15.123:0/3162063689 shutdown_connections 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.205+0000 7f438fe00640 1 --2- 172.21.15.123:0/3162063689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43901050c0 0x7f4390105540 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.206+0000 7f438fe00640 1 --2- 172.21.15.123:0/3162063689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4390102710 0x7f4390104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.206+0000 7f438fe00640 1 --2- 172.21.15.123:0/3162063689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43900ffda0 0x7f43901021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.206+0000 7f438fe00640 1 -- 172.21.15.123:0/3162063689 >> 172.21.15.123:0/3162063689 conn(0x7f43900fb980 msgr2=0x7f43900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.209+0000 7f438fe00640 1 -- 172.21.15.123:0/3162063689 shutdown_connections 2024-03-24T00:44:13.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.209+0000 7f438fe00640 1 -- 172.21.15.123:0/3162063689 wait complete. 2024-03-24T00:44:13.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.213+0000 7f438fe00640 1 Processor -- start 2024-03-24T00:44:13.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.213+0000 7f438fe00640 1 -- start start 2024-03-24T00:44:13.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43900ffda0 0x7f43901955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4390102710 0x7f4390195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390108f90 con 0x7f4390102710 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390108e10 con 0x7f43901050c0 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.215+0000 7f438fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390109110 con 0x7f43900ffda0 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47984/0 (socket says 172.21.15.123:47984) 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 -- 172.21.15.123:0/1037702257 learned_addr learned my addr 172.21.15.123:0/1037702257 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 -- 172.21.15.123:0/1037702257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43900ffda0 msgr2=0x7f43901955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438ea00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43900ffda0 0x7f43901955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43900ffda0 0x7f43901955a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 -- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4390102710 msgr2=0x7f4390195ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4390102710 0x7f4390195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.216+0000 7f438f400640 1 -- 172.21.15.123:0/1037702257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439019d1c0 con 0x7f43901050c0 2024-03-24T00:44:13.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.217+0000 7f438f400640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4388036040 tx=0x7f43880337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.218+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388043020 con 0x7f43901050c0 2024-03-24T00:44:13.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.218+0000 7f438fe00640 1 -- 172.21.15.123:0/1037702257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439019d450 con 0x7f43901050c0 2024-03-24T00:44:13.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.218+0000 7f438fe00640 1 -- 172.21.15.123:0/1037702257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439019da70 con 0x7f43901050c0 2024-03-24T00:44:13.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.218+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43880338a0 con 0x7f43901050c0 2024-03-24T00:44:13.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.218+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388043020 con 0x7f43901050c0 2024-03-24T00:44:13.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.222+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4368005290 con 0x7f43901050c0 2024-03-24T00:44:13.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.228+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4388046020 con 0x7f43901050c0 2024-03-24T00:44:13.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.230+0000 7f4387400640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 0x7f43740a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.233+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4388003b80 con 0x7f43901050c0 2024-03-24T00:44:13.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.234+0000 7f438ea00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 0x7f43740a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.239+0000 7f438ea00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 0x7f43740a4f80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f437c00a810 tx=0x7f437c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.241+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f438804d020 con 0x7f43901050c0 2024-03-24T00:44:13.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.600+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4368005850 con 0x7f43901050c0 2024-03-24T00:44:13.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.601+0000 7f4387400640 1 -- 172.21.15.123:0/1037702257 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f43880bdee0 con 0x7f43901050c0 2024-03-24T00:44:13.602 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:44:13.602 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:44:13.602 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 msgr2=0x7f43740a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 0x7f43740a4f80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f437c00a810 tx=0x7f437c008040 comp rx=0 tx=0).stop 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 msgr2=0x7f439019cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4388036040 tx=0x7f43880337b0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 shutdown_connections 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43901050c0 0x7f439019cb50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f43740a2ac0 0x7f43740a4f80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4390102710 0x7f4390195ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 --2- 172.21.15.123:0/1037702257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43900ffda0 0x7f43901955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 >> 172.21.15.123:0/1037702257 conn(0x7f43900fb980 msgr2=0x7f43900fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 shutdown_connections 2024-03-24T00:44:13.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.603+0000 7f4384c00640 1 -- 172.21.15.123:0/1037702257 wait complete. 2024-03-24T00:44:13.622 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:44:13.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:44:13.685 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:13 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi123 to 176.7M 2024-03-24T00:44:13.685 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:13 smithi123 ceph-mon[31487]: Unable to set osd_memory_target on smithi123 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-24T00:44:13.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:13 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi123 to 176.7M 2024-03-24T00:44:13.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:13 smithi177 ceph-mon[36034]: Unable to set osd_memory_target on smithi123 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-24T00:44:13.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 -- 172.21.15.123:0/2103457396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c104930 msgr2=0x7f5e6c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 --2- 172.21.15.123:0/2103457396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c104930 0x7f5e6c106d20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009900 tx=0x7f5e5802bc50 comp rx=0 tx=0).stop 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 -- 172.21.15.123:0/2103457396 shutdown_connections 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 --2- 172.21.15.123:0/2103457396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c107260 0x7f5e6c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 --2- 172.21.15.123:0/2103457396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c104930 0x7f5e6c106d20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 --2- 172.21.15.123:0/2103457396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c102000 0x7f5e6c1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 -- 172.21.15.123:0/2103457396 >> 172.21.15.123:0/2103457396 conn(0x7f5e6c0fba10 msgr2=0x7f5e6c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 -- 172.21.15.123:0/2103457396 shutdown_connections 2024-03-24T00:44:13.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.811+0000 7f5e6be00640 1 -- 172.21.15.123:0/2103457396 wait complete. 2024-03-24T00:44:13.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.812+0000 7f5e6be00640 1 Processor -- start 2024-03-24T00:44:13.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.812+0000 7f5e6be00640 1 -- start start 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 0x7f5e6c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c107260 0x7f5e6c1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c10d260 con 0x7f5e6c102000 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c10d0e0 con 0x7f5e6c104930 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c10d3e0 con 0x7f5e6c107260 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 0x7f5e6c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:47994/0 (socket says 172.21.15.123:47994) 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 0x7f5e6c199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40294/0 (socket says 172.21.15.123:40294) 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6a000640 1 -- 172.21.15.123:0/201918798 learned_addr learned my addr 172.21.15.123:0/201918798 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.813+0000 7f5e6b400640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c107260 0x7f5e6c1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 -- 172.21.15.123:0/201918798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c107260 msgr2=0x7f5e6c1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c107260 0x7f5e6c1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 -- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 msgr2=0x7f5e6c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 0x7f5e6c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 -- 172.21.15.123:0/201918798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c1a16f0 con 0x7f5e6c104930 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6a000640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5e5802c260 tx=0x7f5e5802c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e70a00640 1 -- 172.21.15.123:0/201918798 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58041070 con 0x7f5e6c104930 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e70a00640 1 -- 172.21.15.123:0/201918798 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e58004450 con 0x7f5e6c104930 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e70a00640 1 -- 172.21.15.123:0/201918798 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58034440 con 0x7f5e6c104930 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c1a1980 con 0x7f5e6c104930 2024-03-24T00:44:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.814+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c1a1e80 con 0x7f5e6c104930 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.817+0000 7f5e74a7d740 1 Processor -- start 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.817+0000 7f5e74a7d740 1 -- start start 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffbbc0 0x555f28ffc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29002e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f28f5f2c0 con 0x555f28ffbbc0 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f28f07c90 con 0x555f28ffa9f0 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f28f5b360 con 0x555f28ffc580 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48010/0 (socket says 172.21.15.123:48010) 2024-03-24T00:44:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 -- 172.21.15.123:0/354593415 learned_addr learned my addr 172.21.15.123:0/354593415 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffbbc0 0x555f28ffc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e70a00640 1 -- 172.21.15.123:0/201918798 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5e58044070 con 0x7f5e6c104930 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 -- 172.21.15.123:0/354593415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 msgr2=0x555f29002e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29002e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 -- 172.21.15.123:0/354593415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffbbc0 msgr2=0x555f28ffc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffbbc0 0x555f28ffc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.818+0000 7f5e6b400640 1 -- 172.21.15.123:0/354593415 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f28f2c080 con 0x555f28ffa9f0 2024-03-24T00:44:13.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.819+0000 7f5e6b400640 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009fd0 tx=0x7f5e640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e65ad2ed1058a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.819+0000 7f5e57e00640 1 -- 172.21.15.123:0/354593415 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64004d20 con 0x555f28ffa9f0 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.819+0000 7f5e57e00640 1 -- 172.21.15.123:0/354593415 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e64004050 con 0x555f28ffa9f0 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.819+0000 7f5e57e00640 1 -- 172.21.15.123:0/354593415 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64005220 con 0x555f28ffa9f0 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- 172.21.15.123:0/354593415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 msgr2=0x555f28ffae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009fd0 tx=0x7f5e640118d0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- 172.21.15.123:0/354593415 shutdown_connections 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29002e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffbbc0 0x555f28ffc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/354593415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffa9f0 0x555f28ffae10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- 172.21.15.123:0/354593415 >> 172.21.15.123:0/354593415 conn(0x555f28ff5de0 msgr2=0x555f28ff61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- 172.21.15.123:0/354593415 shutdown_connections 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- 172.21.15.123:0/354593415 wait complete. 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e70a00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 0x7f5e500a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e70a00640 1 -- 172.21.15.123:0/201918798 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5e580f04d0 con 0x7f5e6c104930 2024-03-24T00:44:13.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 0x7f5e500a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.820+0000 7f5e74a7d740 1 -- start start 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 Processor -- start 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 0x7f5e500a4bd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x555f28ff6d50 tx=0x7f5e5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 0x555f29004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 0x555f2900c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f29003590 con 0x555f28ffc580 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f28f1b0d0 con 0x555f28ffbbc0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f29003710 con 0x555f28ffa9f0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e74a7d740 10 client.0 _ll_register_callbacks cb 0x555f28f1fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 0x555f29004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48020/0 (socket says 172.21.15.123:48020) 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 0x555f29004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58532/0 (socket says 172.21.15.123:58532) 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.821+0000 7f5e6a000640 1 -- 172.21.15.123:0/2267055957 learned_addr learned my addr 172.21.15.123:0/2267055957 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6b400640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 0x555f2900c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 -- 172.21.15.123:0/2267055957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 msgr2=0x555f29004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 0x555f29004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 -- 172.21.15.123:0/2267055957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 msgr2=0x555f2900c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 0x555f2900c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 -- 172.21.15.123:0/2267055957 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f28c733f0 con 0x555f28ffbbc0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6b400640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 0x555f2900c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5e580b9390 tx=0x7f5e580bcdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df766296acaf3a5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e54c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5804b040 con 0x555f28ffbbc0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5e580c5190 con 0x555f28ffbbc0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.822+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e64004050 con 0x555f28ffbbc0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e580da050 con 0x555f28ffbbc0 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 10 client.26252 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555f28d3d7d0 con 0x555f28ffbbc0 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e64004050 con 0x555f28ffbbc0 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e580f0070 con 0x555f28ffbbc0 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 10 client.26252 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 20 client.26252 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f29010660 0x555f29012a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 4 client.26252 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f28f6c800 con 0x555f29010660 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.823+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f29010660 0x555f29012a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.824+0000 7f5e54c00640 10 client.26252 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:13.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.824+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f29010660 0x555f29012a50 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42238b874ed8462 server_cookie=882e91f13226d9e5 in_seq=0 out_seq=0 2024-03-24T00:44:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:13 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi123 to 176.7M 2024-03-24T00:44:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:13 smithi204 ceph-mon[36345]: Unable to set osd_memory_target on smithi123 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e54c00640 1 -- 172.21.15.123:0/2267055957 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555f28f6c800 con 0x555f29010660 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e54c00640 10 client.26252 handle_command_reply: tid=0 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 client.26252 shutdown 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 msgr2=0x555f290051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5e580b9390 tx=0x7f5e580bcdc0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 shutdown_connections 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffc580 0x555f2900c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f290051d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f29010660 0x555f29012a50 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2267055957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffa9f0 0x555f29004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e54c00640 0 client.26252 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 >> 172.21.15.123:0/2267055957 conn(0x555f28ff5de0 msgr2=0x555f28ff5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 shutdown_connections 2024-03-24T00:44:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2267055957 wait complete. 2024-03-24T00:44:13.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.825+0000 7f5e74a7d740 20 client.26252 trim_cache size 0 max 16384 2024-03-24T00:44:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 Processor -- start 2024-03-24T00:44:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 -- start start 2024-03-24T00:44:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f28ff0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffc580 0x555f28ffec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1a8aa0 con 0x555f28ffa9f0 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1ab060 con 0x555f28ffc580 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.839+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1a90a0 con 0x555f28ffbbc0 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58538/0 (socket says 172.21.15.123:58538) 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffc580 0x555f28ffec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 -- 172.21.15.123:0/669894101 learned_addr learned my addr 172.21.15.123:0/669894101 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f28ff0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 -- 172.21.15.123:0/669894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffc580 msgr2=0x555f28ffec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffc580 0x555f28ffec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 -- 172.21.15.123:0/669894101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 msgr2=0x555f28ff0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f28ff0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 -- 172.21.15.123:0/669894101 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f29005e10 con 0x555f28ffbbc0 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f28ff0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e6a000640 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e580bcec0 tx=0x7f5e580bb440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=562cb8454b3a3395 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e57400640 1 -- 172.21.15.123:0/669894101 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5804b070 con 0x555f28ffbbc0 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e57400640 1 -- 172.21.15.123:0/669894101 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e580bcf30 con 0x555f28ffbbc0 2024-03-24T00:44:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.840+0000 7f5e57400640 1 -- 172.21.15.123:0/669894101 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580bc380 con 0x555f28ffbbc0 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 -- 172.21.15.123:0/669894101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 msgr2=0x555f2900c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e580bcec0 tx=0x7f5e580bb440 comp rx=0 tx=0).stop 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 -- 172.21.15.123:0/669894101 shutdown_connections 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffc580 0x555f28ffec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffbbc0 0x555f2900c1d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/669894101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f28ff0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 -- 172.21.15.123:0/669894101 >> 172.21.15.123:0/669894101 conn(0x555f28ff5de0 msgr2=0x555f28ff5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 -- 172.21.15.123:0/669894101 shutdown_connections 2024-03-24T00:44:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.841+0000 7f5e74a7d740 1 -- 172.21.15.123:0/669894101 wait complete. 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 -- start start 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 Processor -- start 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f2900c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29008620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1a96e0 con 0x555f28ffa9f0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1aaa00 con 0x555f28ffbbc0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1a9d40 con 0x555f28ffc580 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.842+0000 7f5e74a7d740 10 client.0 _ll_register_callbacks cb 0x7f5e580b91d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f2900c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40318/0 (socket says 172.21.15.123:40318) 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6aa00640 1 -- 172.21.15.123:0/2181223690 learned_addr learned my addr 172.21.15.123:0/2181223690 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6b400640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 -- 172.21.15.123:0/2181223690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 msgr2=0x555f29008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29008620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 -- 172.21.15.123:0/2181223690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 msgr2=0x555f2900bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 -- 172.21.15.123:0/2181223690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f29008e30 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6b400640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29008620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e6a000640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f2900c0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5e580b9390 tx=0x7f5e58003990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de684088232bf595 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e54c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580a0070 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5e580b8c80 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.843+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e580bcf30 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e5804b050 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 10 client.26258 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555f2906cc30 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e580bcf30 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e580b7aa0 con 0x555f28ffbbc0 2024-03-24T00:44:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 10 client.26258 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 10 client.26258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 20 client.26258 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f2906d160 0x555f2906f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 4 client.26258 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555f28f66b50 con 0x555f2906d160 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.844+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f2906d160 0x555f2906f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e6aa00640 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f2906d160 0x555f2906f580 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75681ecf37ded728 server_cookie=a637d62d9ebfe957 in_seq=0 out_seq=0 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e54c00640 10 client.26258 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e54c00640 1 -- 172.21.15.123:0/2181223690 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555f28f66b50 con 0x555f2906d160 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e54c00640 10 client.26258 handle_command_reply: tid=0 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 client.26258 shutdown 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 msgr2=0x555f2900c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f2900c0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5e580b9390 tx=0x7f5e58003990 comp rx=0 tx=0).stop 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 shutdown_connections 2024-03-24T00:44:13.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555f2906d160 0x555f2906f580 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555f28ffc580 0x555f29008620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555f28ffbbc0 0x555f2900c0c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 --2- 172.21.15.123:0/2181223690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f28ffa9f0 0x555f2900bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 >> 172.21.15.123:0/2181223690 conn(0x555f28ff5de0 msgr2=0x555f28ff9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.845+0000 7f5e54c00640 0 client.26258 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 shutdown_connections 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e74a7d740 1 -- 172.21.15.123:0/2181223690 wait complete. 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e74a7d740 20 client.26258 trim_cache size 0 max 16384 2024-03-24T00:44:13.851 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 msgr2=0x7f5e500a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 0x7f5e500a4bd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x555f28ff6d50 tx=0x7f5e5c005f90 comp rx=0 tx=0).stop 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 msgr2=0x7f5e6c19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5e5802c260 tx=0x7f5e5802c670 comp rx=0 tx=0).stop 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 shutdown_connections 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6c107260 0x7f5e6c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e6c104930 0x7f5e6c19a010 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5e500a2710 0x7f5e500a4bd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 --2- 172.21.15.123:0/201918798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6c102000 0x7f5e6c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:13.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.846+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 >> 172.21.15.123:0/201918798 conn(0x7f5e6c0fba10 msgr2=0x7f5e6c1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:13.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.847+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 shutdown_connections 2024-03-24T00:44:13.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:13.847+0000 7f5e6be00640 1 -- 172.21.15.123:0/201918798 wait complete. 2024-03-24T00:44:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:14 smithi177 ceph-mon[36034]: pgmap v575: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 35 MiB/s wr, 2.94k op/s 2024-03-24T00:44:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:14 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1037702257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:14 smithi204 ceph-mon[36345]: pgmap v575: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 35 MiB/s wr, 2.94k op/s 2024-03-24T00:44:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:14 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1037702257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:14.859 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:14 smithi123 ceph-mon[31487]: pgmap v575: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 35 MiB/s wr, 2.94k op/s 2024-03-24T00:44:14.859 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:14 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1037702257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:14.866 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:44:14.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:14.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:44:15.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 -- 172.21.15.123:0/3929816605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc81028a0 msgr2=0x7fedc8102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 --2- 172.21.15.123:0/3929816605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc81028a0 0x7fedc8102d20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009900 tx=0x7fedb402bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:15.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 -- 172.21.15.123:0/3929816605 shutdown_connections 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 --2- 172.21.15.123:0/3929816605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc8103260 0x7fedc8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 --2- 172.21.15.123:0/3929816605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc81028a0 0x7fedc8102d20 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009900 tx=0x7fedb402bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 --2- 172.21.15.123:0/3929816605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc81016b0 0x7fedc8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 -- 172.21.15.123:0/3929816605 >> 172.21.15.123:0/3929816605 conn(0x7fedc80fcc40 msgr2=0x7fedc80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 -- 172.21.15.123:0/3929816605 shutdown_connections 2024-03-24T00:44:15.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.189+0000 7fedccc00640 1 -- 172.21.15.123:0/3929816605 wait complete. 2024-03-24T00:44:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.190+0000 7fedccc00640 1 Processor -- start 2024-03-24T00:44:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 -- start start 2024-03-24T00:44:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc8103260 0x7fedc8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc8073fd0 0x7fedc8074480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc810d360 con 0x7fedc8073fd0 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc810d1e0 con 0x7fedc81016b0 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedccc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc810d4e0 con 0x7fedc8103260 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48044/0 (socket says 172.21.15.123:48044) 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc5e00640 1 -- 172.21.15.123:0/3600810569 learned_addr learned my addr 172.21.15.123:0/3600810569 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc5400640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc8103260 0x7fedc8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc6800640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc8073fd0 0x7fedc8074480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.191+0000 7fedc5e00640 1 -- 172.21.15.123:0/3600810569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc8103260 msgr2=0x7fedc8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedc5e00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc8103260 0x7fedc8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedc5e00640 1 -- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc8073fd0 msgr2=0x7fedc8074480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedc5e00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc8073fd0 0x7fedc8074480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedc5e00640 1 -- 172.21.15.123:0/3600810569 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc81a1e20 con 0x7fedc81016b0 2024-03-24T00:44:15.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedc5e00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fedb800e9c0 tx=0x7fedb800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedbea00640 1 -- 172.21.15.123:0/3600810569 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb800ccc0 con 0x7fedc81016b0 2024-03-24T00:44:15.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedbea00640 1 -- 172.21.15.123:0/3600810569 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb80035f0 con 0x7fedc81016b0 2024-03-24T00:44:15.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedbea00640 1 -- 172.21.15.123:0/3600810569 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb800ce60 con 0x7fedc81016b0 2024-03-24T00:44:15.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.192+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc81a2110 con 0x7fedc81016b0 2024-03-24T00:44:15.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.193+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc81a2680 con 0x7fedc81016b0 2024-03-24T00:44:15.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.196+0000 7fedbea00640 1 -- 172.21.15.123:0/3600810569 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fedb80037a0 con 0x7fedc81016b0 2024-03-24T00:44:15.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.196+0000 7fedcf992740 1 Processor -- start 2024-03-24T00:44:15.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.196+0000 7fedcf992740 1 -- start start 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f85680590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f855e0eb0 con 0x562f8567b0a0 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f85589a60 con 0x562f85680c00 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedcf992740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f855dcf50 con 0x562f8567ba90 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedc5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedc5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48050/0 (socket says 172.21.15.123:48050) 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedc5400640 1 -- 172.21.15.123:0/3619881918 learned_addr learned my addr 172.21.15.123:0/3619881918 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedc6800640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.197+0000 7fedc5e00640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f85680590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 -- 172.21.15.123:0/3619881918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 msgr2=0x562f85680590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f85680590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 -- 172.21.15.123:0/3619881918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 msgr2=0x562f8567b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 -- 172.21.15.123:0/3619881918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f853bd7e0 con 0x562f85680c00 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc6800640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5e00640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f85680590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedbea00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 0x7fedac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedbea00640 1 -- 172.21.15.123:0/3600810569 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fedb80ced30 con 0x7fedc81016b0 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc6800640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 0x7fedac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.198+0000 7fedc5400640 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fedb4005ab0 tx=0x7fedb4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b34a006eb45dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.199+0000 7fedb3000640 1 -- 172.21.15.123:0/3619881918 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb402c980 con 0x562f85680c00 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.199+0000 7fedb3000640 1 -- 172.21.15.123:0/3619881918 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb402cb20 con 0x562f85680c00 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.199+0000 7fedb3000640 1 -- 172.21.15.123:0/3619881918 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40046c0 con 0x562f85680c00 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- 172.21.15.123:0/3619881918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 msgr2=0x562f85683090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedc6800640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 0x7fedac0a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x562f8567c2f0 tx=0x7fedc0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fedb4005ab0 tx=0x7fedb4009560 comp rx=0 tx=0).stop 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- 172.21.15.123:0/3619881918 shutdown_connections 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f85683090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f85680590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 --2- 172.21.15.123:0/3619881918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- 172.21.15.123:0/3619881918 >> 172.21.15.123:0/3619881918 conn(0x562f85676610 msgr2=0x562f85676a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- 172.21.15.123:0/3619881918 shutdown_connections 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- 172.21.15.123:0/3619881918 wait complete. 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 -- start start 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.200+0000 7fedcf992740 1 Processor -- start 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f85683b50 con 0x562f85680c00 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f8559ceb0 con 0x562f8567b0a0 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f85683cd0 con 0x562f8567ba90 2024-03-24T00:44:15.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedcf992740 10 client.0 _ll_register_callbacks cb 0x562f85591150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48054/0 (socket says 172.21.15.123:48054) 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58590/0 (socket says 172.21.15.123:58590) 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc5e00640 1 -- 172.21.15.123:0/1340641865 learned_addr learned my addr 172.21.15.123:0/1340641865 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.201+0000 7fedc6800640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 -- 172.21.15.123:0/1340641865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 msgr2=0x562f856856d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 -- 172.21.15.123:0/1340641865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 msgr2=0x562f8568c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 -- 172.21.15.123:0/1340641865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f8541df70 con 0x562f8567b0a0 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc6800640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5400640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:15.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedc5e00640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fedb809b3e0 tx=0x7fedb809b570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=541dc7360d8d947d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7feda7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb809b6e0 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fedb8086070 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.202+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedb402cb20 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedb809f830 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 10 client.26270 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562f852b5170 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb402cb20 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fedb80a39f0 con 0x562f8567b0a0 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 10 client.26270 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 20 client.26270 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f85690f10 0x562f85693390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 4 client.26270 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:15.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f855ee3f0 con 0x562f85690f10 2024-03-24T00:44:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.203+0000 7fedc5400640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f85690f10 0x562f85693390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.204+0000 7fedc5400640 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f85690f10 0x562f85693390 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b4ce0a3fb2f672 server_cookie=9438b6dfddfbe055 in_seq=0 out_seq=0 2024-03-24T00:44:15.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.204+0000 7feda7400640 10 client.26270 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7feda7400640 1 -- 172.21.15.123:0/1340641865 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f855ee3f0 con 0x562f85690f10 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7feda7400640 10 client.26270 handle_command_reply: tid=0 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 client.26270 shutdown 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 msgr2=0x562f85685190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fedb809b3e0 tx=0x7fedb809b570 comp rx=0 tx=0).stop 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 shutdown_connections 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f85690f10 0x562f85693390 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f856856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 --2- 172.21.15.123:0/1340641865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f85685190 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 >> 172.21.15.123:0/1340641865 conn(0x562f85676610 msgr2=0x562f85675830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7feda7400640 0 client.26270 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 shutdown_connections 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 1 -- 172.21.15.123:0/1340641865 wait complete. 2024-03-24T00:44:15.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.205+0000 7fedcf992740 20 client.26270 trim_cache size 0 max 16384 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 Processor -- start 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 -- start start 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81a9230 con 0x562f8567b0a0 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81ab7f0 con 0x562f85680c00 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.224+0000 7fedcf992740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81a9830 con 0x562f8567ba90 2024-03-24T00:44:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48066/0 (socket says 172.21.15.123:48066) 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc6800640 1 -- 172.21.15.123:0/2329288054 learned_addr learned my addr 172.21.15.123:0/2329288054 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58592/0 (socket says 172.21.15.123:58592) 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5e00640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 -- 172.21.15.123:0/2329288054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 msgr2=0x562f8567c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 -- 172.21.15.123:0/2329288054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 msgr2=0x562f8567c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 -- 172.21.15.123:0/2329288054 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f8567ac60 con 0x562f8567ba90 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5e00640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc6800640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedc5400640 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedb402c7b0 tx=0x7fedb4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a4dfe43e3e76bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.225+0000 7fedb2600640 1 -- 172.21.15.123:0/2329288054 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb402cc00 con 0x562f8567ba90 2024-03-24T00:44:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedb2600640 1 -- 172.21.15.123:0/2329288054 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb4051a70 con 0x562f8567ba90 2024-03-24T00:44:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedb2600640 1 -- 172.21.15.123:0/2329288054 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb403f9a0 con 0x562f8567ba90 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 -- 172.21.15.123:0/2329288054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 msgr2=0x562f8568c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedb402c7b0 tx=0x7fedb4000bc0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 -- 172.21.15.123:0/2329288054 shutdown_connections 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f85680c00 0x562f8567c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568c6d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 --2- 172.21.15.123:0/2329288054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f8567b0a0 0x562f8567c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 -- 172.21.15.123:0/2329288054 >> 172.21.15.123:0/2329288054 conn(0x562f85676610 msgr2=0x562f8567a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 -- 172.21.15.123:0/2329288054 shutdown_connections 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.226+0000 7fedcf992740 1 -- 172.21.15.123:0/2329288054 wait complete. 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.227+0000 7fedcf992740 1 -- start start 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.227+0000 7fedcf992740 1 Processor -- start 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81a9e70 con 0x562f85680c00 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81ab190 con 0x562f8567b0a0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc81aa4d0 con 0x562f8567ba90 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedcf992740 10 client.0 _ll_register_callbacks cb 0x7fedb8003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48080/0 (socket says 172.21.15.123:48080) 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 -- 172.21.15.123:0/2294470163 learned_addr learned my addr 172.21.15.123:0/2294470163 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5400640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc6800640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 -- 172.21.15.123:0/2294470163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 msgr2=0x562f8568ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 -- 172.21.15.123:0/2294470163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 msgr2=0x562f8568bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 -- 172.21.15.123:0/2294470163 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f8568c130 con 0x562f8567b0a0 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5400640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:15.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7fedc5e00640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fedb8098060 tx=0x7fedb809b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d88b56c9e5ba8de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.228+0000 7feda7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb809b040 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fedb80999e0 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedb4051a70 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedb8097980 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 10 client.26276 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562f856ed0f0 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb4051a70 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fedb80afac0 con 0x562f8567b0a0 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 10 client.26276 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 10 client.26276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 20 client.26276 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f856ed850 0x562f856efc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 4 client.26276 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:44:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.229+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562f855e8740 con 0x562f856ed850 2024-03-24T00:44:15.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.230+0000 7fedc5400640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f856ed850 0x562f856efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.230+0000 7fedc5400640 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f856ed850 0x562f856efc70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f569947bf7842db6 server_cookie=7b939b3c411c9bfe in_seq=0 out_seq=0 2024-03-24T00:44:15.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.230+0000 7feda7400640 10 client.26276 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.231+0000 7feda7400640 1 -- 172.21.15.123:0/2294470163 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562f855e8740 con 0x562f856ed850 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.231+0000 7feda7400640 10 client.26276 handle_command_reply: tid=0 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.231+0000 7fedcf992740 1 client.26276 shutdown 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 msgr2=0x562f8568a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fedb8098060 tx=0x7fedb809b6e0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 shutdown_connections 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562f85680c00 0x562f8568bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x562f856ed850 0x562f856efc70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562f8567ba90 0x562f8568ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 --2- 172.21.15.123:0/2294470163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f8567b0a0 0x562f8568a5f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 >> 172.21.15.123:0/2294470163 conn(0x562f85676610 msgr2=0x562f85674fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7feda7400640 0 client.26276 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 shutdown_connections 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 1 -- 172.21.15.123:0/2294470163 wait complete. 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.232+0000 7fedcf992740 20 client.26276 trim_cache size 0 max 16384 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "d2c2d534-2512-4eb2-9665-1a80a9bcd2d0", 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:44:15.237 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:15.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 msgr2=0x7fedac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 0x7fedac0a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x562f8567c2f0 tx=0x7fedc0009500 comp rx=0 tx=0).stop 2024-03-24T00:44:15.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 msgr2=0x7fedc8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fedb800e9c0 tx=0x7fedb800edc0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 shutdown_connections 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fedac0a2880 0x7fedac0a4d40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedc8073fd0 0x7fedc8074480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedc8103260 0x7fedc8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 --2- 172.21.15.123:0/3600810569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc81016b0 0x7fedc8075450 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 >> 172.21.15.123:0/3600810569 conn(0x7fedc80fcc40 msgr2=0x7fedc8105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 shutdown_connections 2024-03-24T00:44:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.233+0000 7fedccc00640 1 -- 172.21.15.123:0/3600810569 wait complete. 2024-03-24T00:44:15.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:44:15.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:44:15.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 -- 172.21.15.123:0/2198427936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d01037c0 msgr2=0x7f67d0103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 --2- 172.21.15.123:0/2198427936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d01037c0 0x7f67d0103be0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009900 tx=0x7f67c002bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 -- 172.21.15.123:0/2198427936 shutdown_connections 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 --2- 172.21.15.123:0/2198427936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d0105370 0x7f67d010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 --2- 172.21.15.123:0/2198427936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01049b0 0x7f67d0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 --2- 172.21.15.123:0/2198427936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d01037c0 0x7f67d0103be0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 -- 172.21.15.123:0/2198427936 >> 172.21.15.123:0/2198427936 conn(0x7f67d00fed50 msgr2=0x7f67d0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 -- 172.21.15.123:0/2198427936 shutdown_connections 2024-03-24T00:44:15.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.442+0000 7f67d4a00640 1 -- 172.21.15.123:0/2198427936 wait complete. 2024-03-24T00:44:15.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.443+0000 7f67d4a00640 1 Processor -- start 2024-03-24T00:44:15.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.443+0000 7f67d4a00640 1 -- start start 2024-03-24T00:44:15.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d01049b0 0x7f67d0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d0105370 0x7f67d01a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d010f460 con 0x7f67d01037c0 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d010f2e0 con 0x7f67d0105370 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67d4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d010f5e0 con 0x7f67d01049b0 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40398/0 (socket says 172.21.15.123:40398) 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d0105370 0x7f67d01a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 -- 172.21.15.123:0/1271704051 learned_addr learned my addr 172.21.15.123:0/1271704051 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cea00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d01049b0 0x7f67d0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 -- 172.21.15.123:0/1271704051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d01049b0 msgr2=0x7f67d0199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d01049b0 0x7f67d0199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 -- 172.21.15.123:0/1271704051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d0105370 msgr2=0x7f67d01a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d0105370 0x7f67d01a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.444+0000 7f67cf400640 1 -- 172.21.15.123:0/1271704051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d01a16d0 con 0x7f67d01037c0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67cf400640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67c0000c00 tx=0x7f67c002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67c7e00640 1 -- 172.21.15.123:0/1271704051 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0043020 con 0x7f67d01037c0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67c7e00640 1 -- 172.21.15.123:0/1271704051 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67c00054e0 con 0x7f67d01037c0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d01a1960 con 0x7f67d01037c0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67c7e00640 1 -- 172.21.15.123:0/1271704051 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0003c40 con 0x7f67d01037c0 2024-03-24T00:44:15.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.445+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d01a1f00 con 0x7f67d01037c0 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.448+0000 7f67d8db5740 1 Processor -- start 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.448+0000 7f67c7e00640 1 -- 172.21.15.123:0/1271704051 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f67c0046020 con 0x7f67d01037c0 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.448+0000 7f67d8db5740 1 -- start start 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aabf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 0x55999aac2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999aa1f250 con 0x55999aabfbb0 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999a9c7a30 con 0x55999aaba050 2024-03-24T00:44:15.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67d8db5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999aa1b2f0 con 0x55999aabaa40 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48100/0 (socket says 172.21.15.123:48100) 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aabf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 -- 172.21.15.123:0/3830026454 learned_addr learned my addr 172.21.15.123:0/3830026454 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cea00640 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 0x55999aac2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 -- 172.21.15.123:0/3830026454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 msgr2=0x55999aabf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aabf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 -- 172.21.15.123:0/3830026454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 msgr2=0x55999aac2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 0x55999aac2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cfe00640 1 -- 172.21.15.123:0/3830026454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55999a9b2690 con 0x55999aaba050 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.449+0000 7f67cea00640 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 0x55999aac2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009fd0 tx=0x7f67c80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d59f3201b69ee631 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67c7e00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67bbe00640 1 -- 172.21.15.123:0/3830026454 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8004d20 con 0x55999aaba050 2024-03-24T00:44:15.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67bbe00640 1 -- 172.21.15.123:0/3830026454 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67c8004050 con 0x55999aaba050 2024-03-24T00:44:15.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67bbe00640 1 -- 172.21.15.123:0/3830026454 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8005260 con 0x55999aaba050 2024-03-24T00:44:15.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67c7e00640 1 -- 172.21.15.123:0/1271704051 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67c00f09b0 con 0x7f67d01037c0 2024-03-24T00:44:15.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.450+0000 7f67cf400640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- 172.21.15.123:0/3830026454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 msgr2=0x55999aaba470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009fd0 tx=0x7f67c80118d0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67cf400640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 0x7f67b40a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f67c00ee830 tx=0x7f67c00dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- 172.21.15.123:0/3830026454 shutdown_connections 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabfbb0 0x55999aac2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aabf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3830026454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aaba470 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- 172.21.15.123:0/3830026454 >> 172.21.15.123:0/3830026454 conn(0x55999aa82200 msgr2=0x55999aab59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- 172.21.15.123:0/3830026454 shutdown_connections 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- 172.21.15.123:0/3830026454 wait complete. 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.451+0000 7f67d8db5740 1 -- start start 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 Processor -- start 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 0x55999aac4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999aac2b60 con 0x55999aabaa40 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999a9dae80 con 0x55999aaba050 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999aac2ce0 con 0x55999aabfbb0 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.452+0000 7f67d8db5740 10 client.0 _ll_register_callbacks cb 0x55999a9ed130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 0x55999aac4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48114/0 (socket says 172.21.15.123:48114) 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cfe00640 1 -- 172.21.15.123:0/3796212766 learned_addr learned my addr 172.21.15.123:0/3796212766 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 -- 172.21.15.123:0/3796212766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 msgr2=0x55999aacb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 -- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 msgr2=0x55999aac4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 -- 172.21.15.123:0/3796212766 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55999a7fd7d0 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67cea00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 0x55999aac4590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f67d019afc0 tx=0x7f67bc012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba7f1cccf6243 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.453+0000 7f67b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc004050 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67bc0041f0 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67c8004050 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67bc004430 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67d8db5740 10 client.16122 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55999a7333f0 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67c8004050 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f67bc003b50 con 0x55999aabaa40 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67d8db5740 10 client.16122 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.454+0000 7f67d8db5740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:15.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67d8db5740 20 client.16122 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999aacfef0 0x55999aad2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67d8db5740 4 client.16122 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55999aa2c790 con 0x55999aacfef0 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999aacfef0 0x55999aad2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67cfe00640 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999aacfef0 0x55999aad2330 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51c0835424db4cbc server_cookie=63d5f4bbd8165f59 in_seq=0 out_seq=0 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.455+0000 7f67b8c00640 10 client.16122 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67b8c00640 1 -- 172.21.15.123:0/3796212766 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55999aa2c790 con 0x55999aacfef0 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67b8c00640 10 client.16122 handle_command_reply: tid=0 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 client.16122 shutdown 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 msgr2=0x55999aac4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 0x55999aac4590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f67d019afc0 tx=0x7f67bc012b60 comp rx=0 tx=0).stop 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 shutdown_connections 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999aacfef0 0x55999aad2330 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aabaa40 0x55999aac4590 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 --2- 172.21.15.123:0/3796212766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aaba050 0x55999aac4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67b8c00640 0 client.16122 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 >> 172.21.15.123:0/3796212766 conn(0x55999aa82200 msgr2=0x55999aab4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 shutdown_connections 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 1 -- 172.21.15.123:0/3796212766 wait complete. 2024-03-24T00:44:15.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.456+0000 7f67d8db5740 20 client.16122 trim_cache size 0 max 16384 2024-03-24T00:44:15.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.474+0000 7f67d8db5740 1 Processor -- start 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.474+0000 7f67d8db5740 1 -- start start 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aabf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabfbb0 0x55999aab8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01a8be0 con 0x55999aaba050 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01ab1a0 con 0x55999aabfbb0 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67d8db5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01a91e0 con 0x55999aabaa40 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58658/0 (socket says 172.21.15.123:58658) 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 -- 172.21.15.123:0/2377402729 learned_addr learned my addr 172.21.15.123:0/2377402729 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cfe00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aabf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cfe00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabfbb0 0x55999aab8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 -- 172.21.15.123:0/2377402729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabfbb0 msgr2=0x55999aab8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabfbb0 0x55999aab8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 -- 172.21.15.123:0/2377402729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 msgr2=0x55999aabf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aabf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 -- 172.21.15.123:0/2377402729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55999a6f30f0 con 0x55999aabaa40 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cfe00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aabf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.475+0000 7f67cea00640 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f67bc010500 tx=0x7f67bc010880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfdbd9d396079756 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.476+0000 7f67bb400640 1 -- 172.21.15.123:0/2377402729 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc003f90 con 0x55999aabaa40 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.476+0000 7f67bb400640 1 -- 172.21.15.123:0/2377402729 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67bc010e50 con 0x55999aabaa40 2024-03-24T00:44:15.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.476+0000 7f67bb400640 1 -- 172.21.15.123:0/2377402729 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00fdd0 con 0x55999aabaa40 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.476+0000 7f67d8db5740 1 -- 172.21.15.123:0/2377402729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 msgr2=0x55999aacb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.476+0000 7f67d8db5740 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f67bc010500 tx=0x7f67bc010880 comp rx=0 tx=0).stop 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 -- 172.21.15.123:0/2377402729 shutdown_connections 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabfbb0 0x55999aab8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabaa40 0x55999aacb590 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 --2- 172.21.15.123:0/2377402729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aabf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 -- 172.21.15.123:0/2377402729 >> 172.21.15.123:0/2377402729 conn(0x55999aa82200 msgr2=0x55999aabbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 -- 172.21.15.123:0/2377402729 shutdown_connections 2024-03-24T00:44:15.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 -- 172.21.15.123:0/2377402729 wait complete. 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 -- start start 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.477+0000 7f67d8db5740 1 Processor -- start 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aac9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01a9820 con 0x55999aaba050 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01aab40 con 0x55999aabaa40 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01a9e80 con 0x55999aabfbb0 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67d8db5740 10 client.0 _ll_register_callbacks cb 0x7f67bc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aac9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:48140/0 (socket says 172.21.15.123:48140) 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 -- 172.21.15.123:0/1598069313 learned_addr learned my addr 172.21.15.123:0/1598069313 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:15.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cfe00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 -- 172.21.15.123:0/1598069313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 msgr2=0x55999aacaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 -- 172.21.15.123:0/1598069313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 msgr2=0x55999aac9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aac9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 -- 172.21.15.123:0/1598069313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55999aacb1d0 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.478+0000 7f67cea00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67bc012ba0 tx=0x7f67bc009df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd90e73432362670 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc035070 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f67bc0108f0 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67bc010e50 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67bc0041a0 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67d8db5740 10 client.26294 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55999ab2c190 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.479+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67bc010e50 con 0x55999aabaa40 2024-03-24T00:44:15.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f67bc0053d0 con 0x55999aabaa40 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 10 client.26294 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 20 client.26294 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999ab2cb20 0x55999ab2ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 4 client.26294 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55999aa26ae0 con 0x55999ab2cb20 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67cfe00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999ab2cb20 0x55999ab2ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67cfe00640 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999ab2cb20 0x55999ab2ef40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c2854abc6fcdc41 server_cookie=ea288b82ef201f31 in_seq=0 out_seq=0 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67b8c00640 10 client.26294 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67b8c00640 1 -- 172.21.15.123:0/1598069313 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55999aa26ae0 con 0x55999ab2cb20 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67b8c00640 10 client.26294 handle_command_reply: tid=0 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.480+0000 7f67d8db5740 1 client.26294 shutdown 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 msgr2=0x55999aac9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67bc012ba0 tx=0x7f67bc009df0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 shutdown_connections 2024-03-24T00:44:15.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55999ab2cb20 0x55999ab2ef40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55999aabfbb0 0x55999aacaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999aabaa40 0x55999aac9c60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 --2- 172.21.15.123:0/1598069313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55999aaba050 0x55999aac9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 >> 172.21.15.123:0/1598069313 conn(0x55999aa82200 msgr2=0x55999aab4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67b8c00640 0 client.26294 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 shutdown_connections 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 1 -- 172.21.15.123:0/1598069313 wait complete. 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.481+0000 7f67d8db5740 20 client.26294 trim_cache size 0 max 16384 2024-03-24T00:44:15.486 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "d2c2d534-2512-4eb2-9665-1a80a9bcd2d0": { 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "d2c2d534-2512-4eb2-9665-1a80a9bcd2d0", 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:44:15.487 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 msgr2=0x7f67b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 0x7f67b40a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f67c00ee830 tx=0x7f67c00dd040 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 msgr2=0x7f67d0199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67c0000c00 tx=0x7f67c002c650 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 shutdown_connections 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d0105370 0x7f67d01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67d01049b0 0x7f67d0199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 --2- 172.21.15.123:0/1271704051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d01037c0 0x7f67d0199ab0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 >> 172.21.15.123:0/1271704051 conn(0x7f67d00fed50 msgr2=0x7f67d0107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 shutdown_connections 2024-03-24T00:44:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:15.482+0000 7f67d4a00640 1 -- 172.21.15.123:0/1271704051 wait complete. 2024-03-24T00:44:15.498 INFO:tasks.cephfs.filesystem:scrub status for tag:d2c2d534-2512-4eb2-9665-1a80a9bcd2d0 - {'path': '/', 'tag': 'd2c2d534-2512-4eb2-9665-1a80a9bcd2d0', 'options': 'recursive,force'} 2024-03-24T00:44:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:16 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:44:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:16 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:16.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:16 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:44:16.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:16 smithi177 ceph-mon[36034]: pgmap v576: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 3.39k op/s 2024-03-24T00:44:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:16 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:44:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:16 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:16 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:44:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:16 smithi204 ceph-mon[36345]: pgmap v576: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 3.39k op/s 2024-03-24T00:44:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:16 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:44:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:16 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:16 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:44:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:16 smithi123 ceph-mon[31487]: pgmap v576: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 MiB/s wr, 3.39k op/s 2024-03-24T00:44:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:17 smithi204 ceph-mon[36345]: pgmap v577: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 34 MiB/s wr, 3.15k op/s 2024-03-24T00:44:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:17 smithi123 ceph-mon[31487]: pgmap v577: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 34 MiB/s wr, 3.15k op/s 2024-03-24T00:44:17.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:17 smithi177 ceph-mon[36034]: pgmap v577: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 34 MiB/s wr, 3.15k op/s 2024-03-24T00:44:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:18 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:18 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:18 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:19.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:19 smithi204 ceph-mon[36345]: pgmap v578: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 38 MiB/s wr, 3.65k op/s 2024-03-24T00:44:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:19 smithi123 ceph-mon[31487]: pgmap v578: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 38 MiB/s wr, 3.65k op/s 2024-03-24T00:44:19.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:19 smithi177 ceph-mon[36034]: pgmap v578: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 38 MiB/s wr, 3.65k op/s 2024-03-24T00:44:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:22 smithi177 ceph-mon[36034]: pgmap v579: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 38 MiB/s wr, 3.92k op/s 2024-03-24T00:44:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:22 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:44:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:22 smithi204 ceph-mon[36345]: pgmap v579: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 38 MiB/s wr, 3.92k op/s 2024-03-24T00:44:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:22 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:44:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:22 smithi123 ceph-mon[31487]: pgmap v579: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 38 MiB/s wr, 3.92k op/s 2024-03-24T00:44:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:22 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:44:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 -- 172.21.15.123:0/582114886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741048a0 msgr2=0x7fc274106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 --2- 172.21.15.123:0/582114886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741048a0 0x7fc274106c90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009900 tx=0x7fc25c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 -- 172.21.15.123:0/582114886 shutdown_connections 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 --2- 172.21.15.123:0/582114886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741071d0 0x7fc274109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 --2- 172.21.15.123:0/582114886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741048a0 0x7fc274106c90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 --2- 172.21.15.123:0/582114886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 -- 172.21.15.123:0/582114886 >> 172.21.15.123:0/582114886 conn(0x7fc2740fb980 msgr2=0x7fc2740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 -- 172.21.15.123:0/582114886 shutdown_connections 2024-03-24T00:44:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.975+0000 7fc273e00640 1 -- 172.21.15.123:0/582114886 wait complete. 2024-03-24T00:44:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.976+0000 7fc273e00640 1 Processor -- start 2024-03-24T00:44:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.976+0000 7fc273e00640 1 -- start start 2024-03-24T00:44:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741048a0 0x7fc274199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 0x7fc2741a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27410d190 con 0x7fc2741048a0 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27410d010 con 0x7fc2741071d0 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27410d310 con 0x7fc274101f70 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc273400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 0x7fc2741a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56744/0 (socket says 172.21.15.123:56744) 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc272a00640 1 -- 172.21.15.123:0/736688783 learned_addr learned my addr 172.21.15.123:0/736688783 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.977+0000 7fc26be00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741048a0 0x7fc274199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 -- 172.21.15.123:0/736688783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 msgr2=0x7fc274199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 -- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741048a0 msgr2=0x7fc274199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741048a0 0x7fc274199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 -- 172.21.15.123:0/736688783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2741a1640 con 0x7fc2741071d0 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273400640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 0x7fc2741a0fd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009fd0 tx=0x7fc26c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc26b400640 1 -- 172.21.15.123:0/736688783 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c00ce20 con 0x7fc2741071d0 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2741a1930 con 0x7fc2741071d0 2024-03-24T00:44:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc26b400640 1 -- 172.21.15.123:0/736688783 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc26c005020 con 0x7fc2741071d0 2024-03-24T00:44:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc26b400640 1 -- 172.21.15.123:0/736688783 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c010720 con 0x7fc2741071d0 2024-03-24T00:44:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.978+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741a1e00 con 0x7fc2741071d0 2024-03-24T00:44:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.982+0000 7fc27c3e7740 1 Processor -- start 2024-03-24T00:44:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc26b400640 1 -- 172.21.15.123:0/736688783 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc26c005540 con 0x7fc2741071d0 2024-03-24T00:44:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 -- start start 2024-03-24T00:44:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d592932f50 0x55d5929353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5928f03b0 con 0x55d59292d3f0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d592892830 con 0x55d592932f50 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.983+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d59283b5d0 con 0x55d59292dde0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc26be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56752/0 (socket says 172.21.15.123:56752) 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 -- 172.21.15.123:0/3178887296 learned_addr learned my addr 172.21.15.123:0/3178887296 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc26be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48672/0 (socket says 172.21.15.123:48672) 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc26be00640 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d592932f50 0x55d5929353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 -- 172.21.15.123:0/3178887296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d592932f50 msgr2=0x55d5929353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d592932f50 0x55d5929353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 -- 172.21.15.123:0/3178887296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 msgr2=0x55d59292d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 -- 172.21.15.123:0/3178887296 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5928532c0 con 0x55d59292dde0 2024-03-24T00:44:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc26be00640 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc272a00640 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc264009e10 tx=0x7fc264011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b9dbe5f94088fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.984+0000 7fc26b400640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 0x7fc2580a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.985+0000 7fc26b400640 1 -- 172.21.15.123:0/736688783 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc26c0cf240 con 0x7fc2741071d0 2024-03-24T00:44:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.985+0000 7fc263400640 1 -- 172.21.15.123:0/3178887296 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26401e070 con 0x55d59292dde0 2024-03-24T00:44:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.985+0000 7fc26be00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 0x7fc2580a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.985+0000 7fc263400640 1 -- 172.21.15.123:0/3178887296 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc264019b10 con 0x55d59292dde0 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.985+0000 7fc263400640 1 -- 172.21.15.123:0/3178887296 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26400fba0 con 0x55d59292dde0 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3178887296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 msgr2=0x55d5929328e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc26be00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 0x7fc2580a4cb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc25c02c170 tx=0x7fc25c02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc264009e10 tx=0x7fc264011810 comp rx=0 tx=0).stop 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3178887296 shutdown_connections 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d592932f50 0x55d5929353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d59292dde0 0x55d5929328e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3178887296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59292d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3178887296 >> 172.21.15.123:0/3178887296 conn(0x55d592928960 msgr2=0x55d592928d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3178887296 shutdown_connections 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.986+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3178887296 wait complete. 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 -- start start 2024-03-24T00:44:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 Processor -- start 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d592937570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 0x55d59293eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d59284ea20 con 0x55d59292d3f0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d59288e8d0 con 0x55d59292dde0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d592935f00 con 0x55d592932f50 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.987+0000 7fc27c3e7740 10 client.0 _ll_register_callbacks cb 0x55d59288b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49518/0 (socket says 172.21.15.123:49518) 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc273400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 0x55d59293eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 -- 172.21.15.123:0/3158809691 learned_addr learned my addr 172.21.15.123:0/3158809691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc272a00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d592937570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 -- 172.21.15.123:0/3158809691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 msgr2=0x55d59293eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 0x55d59293eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 -- 172.21.15.123:0/3158809691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 msgr2=0x55d592937570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d592937570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 -- 172.21.15.123:0/3158809691 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5925a7210 con 0x55d59292dde0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc273400640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 0x55d59293eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc272a00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d592937570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc26be00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771992a683715a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.988+0000 7fc257e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c04e460 con 0x55d59292dde0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc25c04e600 con 0x55d59292dde0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc264019b10 con 0x55d59292dde0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc25c04e800 con 0x55d59292dde0 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc27c3e7740 10 client.26306 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d592673620 con 0x55d59292dde0 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc264019b10 con 0x55d59292dde0 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.989+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc25c040040 con 0x55d59292dde0 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 10 client.26306 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 10 client.26306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 20 client.26306 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d5929433e0 0x55d592945810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 4 client.26306 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d59289fd70 con 0x55d5929433e0 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.990+0000 7fc272a00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d5929433e0 0x55d592945810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.991+0000 7fc272a00640 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d5929433e0 0x55d592945810 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a1f1694958cbce server_cookie=d99fe7803bb31c5d in_seq=0 out_seq=0 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.991+0000 7fc257e00640 10 client.26306 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.991+0000 7fc257e00640 1 -- 172.21.15.123:0/3158809691 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d59289fd70 con 0x55d5929433e0 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc257e00640 10 client.26306 handle_command_reply: tid=0 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 client.26306 shutdown 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 msgr2=0x55d592937ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 shutdown_connections 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d5929433e0 0x55d592945810 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592932f50 0x55d59293eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d592937ab0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc257e00640 0 client.26306 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/3158809691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d592937570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.992+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 >> 172.21.15.123:0/3158809691 conn(0x55d592928960 msgr2=0x55d592927b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.993+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 shutdown_connections 2024-03-24T00:44:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.993+0000 7fc27c3e7740 1 -- 172.21.15.123:0/3158809691 wait complete. 2024-03-24T00:44:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:23.993+0000 7fc27c3e7740 20 client.26306 trim_cache size 0 max 16384 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 Processor -- start 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 -- start start 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d5929236a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292dde0 0x55d59292f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a8a30 con 0x55d59292dde0 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741aaff0 con 0x55d59292d3f0 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a9030 con 0x55d592928960 2024-03-24T00:44:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d5929236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc26be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc26be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49532/0 (socket says 172.21.15.123:49532) 2024-03-24T00:44:24.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc26be00640 1 -- 172.21.15.123:0/19734964 learned_addr learned my addr 172.21.15.123:0/19734964 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:24.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc272a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d5929236a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56770/0 (socket says 172.21.15.123:56770) 2024-03-24T00:44:24.012 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.011+0000 7fc273400640 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292dde0 0x55d59292f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 -- 172.21.15.123:0/19734964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 msgr2=0x55d5929236a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:24.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d5929236a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.013 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 -- 172.21.15.123:0/19734964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292dde0 msgr2=0x55d59292f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292dde0 0x55d59292f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 -- 172.21.15.123:0/19734964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d592937e20 con 0x55d59292d3f0 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc26be00640 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0429e0 tx=0x7fc25c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=198ec043efa6bb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc262a00640 1 -- 172.21.15.123:0/19734964 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c02ca40 con 0x55d59292d3f0 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc262a00640 1 -- 172.21.15.123:0/19734964 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc25c053070 con 0x55d59292d3f0 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.012+0000 7fc262a00640 1 -- 172.21.15.123:0/19734964 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c04ec90 con 0x55d59292d3f0 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 -- 172.21.15.123:0/19734964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 msgr2=0x55d59293eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0429e0 tx=0x7fc25c009780 comp rx=0 tx=0).stop 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 -- 172.21.15.123:0/19734964 shutdown_connections 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292dde0 0x55d59292f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292d3f0 0x55d59293eab0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/19734964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d5929236a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 -- 172.21.15.123:0/19734964 >> 172.21.15.123:0/19734964 conn(0x55d592932a50 msgr2=0x55d592928d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 -- 172.21.15.123:0/19734964 shutdown_connections 2024-03-24T00:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.013+0000 7fc27c3e7740 1 -- 172.21.15.123:0/19734964 wait complete. 2024-03-24T00:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.014+0000 7fc27c3e7740 1 -- start start 2024-03-24T00:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.014+0000 7fc27c3e7740 1 Processor -- start 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d59293c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d59293d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a9670 con 0x55d59292d3f0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741aa990 con 0x55d59292dde0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a9cd0 con 0x55d592928960 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc27c3e7740 10 client.0 _ll_register_callbacks cb 0x7fc26c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc26be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc26be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48698/0 (socket says 172.21.15.123:48698) 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc26be00640 1 -- 172.21.15.123:0/640155106 learned_addr learned my addr 172.21.15.123:0/640155106 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc273400640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d59293d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc272a00640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d59293c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc273400640 1 -- 172.21.15.123:0/640155106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 msgr2=0x55d59293c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc273400640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d59293c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc273400640 1 -- 172.21.15.123:0/640155106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 msgr2=0x55d59293c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.015+0000 7fc273400640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc273400640 1 -- 172.21.15.123:0/640155106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d59293dc30 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc26be00640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc272a00640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d59293c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc273400640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d59293d640 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc26c09b890 tx=0x7fc26c097eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5edbad06d50c00ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc257e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c0a4330 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc26c0ba1e0 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc25c053070 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc26c0ba3e0 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc27c3e7740 10 client.26312 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.016+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d59299ece0 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc25c053070 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc26c0a0c00 con 0x55d59292dde0 2024-03-24T00:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 10 client.26312 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 10 client.26312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 20 client.26312 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d592932f50 0x55d5929a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 4 client.26312 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d59289a0c0 con 0x55d592932f50 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.017+0000 7fc272a00640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d592932f50 0x55d5929a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.018+0000 7fc272a00640 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d592932f50 0x55d5929a16b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d45a752c2c1ecd07 server_cookie=5cf7a8489ab56610 in_seq=0 out_seq=0 2024-03-24T00:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:24.018+0000 7fc257e00640 10 client.26312 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:24 smithi177 ceph-mon[36034]: pgmap v580: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 36 MiB/s wr, 3.75k op/s 2024-03-24T00:44:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:24 smithi204 ceph-mon[36345]: pgmap v580: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 36 MiB/s wr, 3.75k op/s 2024-03-24T00:44:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:24 smithi123 ceph-mon[31487]: pgmap v580: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 36 MiB/s wr, 3.75k op/s 2024-03-24T00:44:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:26 smithi177 ceph-mon[36034]: pgmap v581: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 41 MiB/s wr, 4.27k op/s 2024-03-24T00:44:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:26 smithi204 ceph-mon[36345]: pgmap v581: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 41 MiB/s wr, 4.27k op/s 2024-03-24T00:44:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:26 smithi123 ceph-mon[31487]: pgmap v581: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 41 MiB/s wr, 4.27k op/s 2024-03-24T00:44:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:28 smithi177 ceph-mon[36034]: pgmap v582: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 34 MiB/s wr, 3.52k op/s 2024-03-24T00:44:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:28 smithi204 ceph-mon[36345]: pgmap v582: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 34 MiB/s wr, 3.52k op/s 2024-03-24T00:44:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:28 smithi123 ceph-mon[31487]: pgmap v582: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 34 MiB/s wr, 3.52k op/s 2024-03-24T00:44:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:30 smithi177 ceph-mon[36034]: pgmap v583: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 36 MiB/s wr, 3.70k op/s 2024-03-24T00:44:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:30 smithi204 ceph-mon[36345]: pgmap v583: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 36 MiB/s wr, 3.70k op/s 2024-03-24T00:44:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:30 smithi123 ceph-mon[31487]: pgmap v583: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 36 MiB/s wr, 3.70k op/s 2024-03-24T00:44:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:32 smithi177 ceph-mon[36034]: pgmap v584: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 37 MiB/s wr, 3.87k op/s 2024-03-24T00:44:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:32 smithi204 ceph-mon[36345]: pgmap v584: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 37 MiB/s wr, 3.87k op/s 2024-03-24T00:44:32.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:32 smithi123 ceph-mon[31487]: pgmap v584: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 37 MiB/s wr, 3.87k op/s 2024-03-24T00:44:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:34 smithi177 ceph-mon[36034]: pgmap v585: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:44:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:34 smithi204 ceph-mon[36345]: pgmap v585: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:44:34.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:34 smithi123 ceph-mon[31487]: pgmap v585: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:44:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:36 smithi177 ceph-mon[36034]: pgmap v586: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 37 MiB/s wr, 3.74k op/s 2024-03-24T00:44:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:36 smithi204 ceph-mon[36345]: pgmap v586: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 37 MiB/s wr, 3.74k op/s 2024-03-24T00:44:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:36 smithi123 ceph-mon[31487]: pgmap v586: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 37 MiB/s wr, 3.74k op/s 2024-03-24T00:44:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:38 smithi177 ceph-mon[36034]: pgmap v587: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 33 MiB/s wr, 3.33k op/s 2024-03-24T00:44:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:38 smithi204 ceph-mon[36345]: pgmap v587: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 33 MiB/s wr, 3.33k op/s 2024-03-24T00:44:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:38 smithi123 ceph-mon[31487]: pgmap v587: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 33 MiB/s wr, 3.33k op/s 2024-03-24T00:44:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:40 smithi177 ceph-mon[36034]: pgmap v588: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.60k op/s 2024-03-24T00:44:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:40 smithi204 ceph-mon[36345]: pgmap v588: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.60k op/s 2024-03-24T00:44:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:40 smithi123 ceph-mon[31487]: pgmap v588: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 35 MiB/s wr, 3.60k op/s 2024-03-24T00:44:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:42 smithi177 ceph-mon[36034]: pgmap v589: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 41 MiB/s wr, 4.08k op/s 2024-03-24T00:44:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:42 smithi204 ceph-mon[36345]: pgmap v589: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 41 MiB/s wr, 4.08k op/s 2024-03-24T00:44:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:42 smithi123 ceph-mon[31487]: pgmap v589: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 41 MiB/s wr, 4.08k op/s 2024-03-24T00:44:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:44 smithi177 ceph-mon[36034]: pgmap v590: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:44 smithi204 ceph-mon[36345]: pgmap v590: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:44 smithi123 ceph-mon[31487]: pgmap v590: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:45.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:44:45.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:44:45.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.898+0000 7fc073400640 1 -- 172.21.15.123:0/1543505372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 msgr2=0x7fc06c10ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.898+0000 7fc073400640 1 --2- 172.21.15.123:0/1543505372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c10ad00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc0600074e0 tx=0x7fc06002c8b0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 -- 172.21.15.123:0/1543505372 shutdown_connections 2024-03-24T00:44:45.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 --2- 172.21.15.123:0/1543505372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 --2- 172.21.15.123:0/1543505372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 0x7fc06c10d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 --2- 172.21.15.123:0/1543505372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c10ad00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 -- 172.21.15.123:0/1543505372 >> 172.21.15.123:0/1543505372 conn(0x7fc06c06e500 msgr2=0x7fc06c070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:45.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 -- 172.21.15.123:0/1543505372 shutdown_connections 2024-03-24T00:44:45.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.899+0000 7fc073400640 1 -- 172.21.15.123:0/1543505372 wait complete. 2024-03-24T00:44:45.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.903+0000 7fc073400640 1 Processor -- start 2024-03-24T00:44:45.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.904+0000 7fc073400640 1 -- start start 2024-03-24T00:44:45.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.905+0000 7fc073400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c1a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.906+0000 7fc073400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 0x7fc06c1a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.907+0000 7fc070a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c1a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.907+0000 7fc070a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c1a67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49970/0 (socket says 172.21.15.123:49970) 2024-03-24T00:44:45.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.908+0000 7fc070a00640 1 -- 172.21.15.123:0/3089188168 learned_addr learned my addr 172.21.15.123:0/3089188168 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:45.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.908+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c1adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.909+0000 7fc071400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c1adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.909+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1133f0 con 0x7fc06c10b2d0 2024-03-24T00:44:45.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.909+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c113270 con 0x7fc06c108910 2024-03-24T00:44:45.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.909+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c113570 con 0x7fc06c10dd70 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.910+0000 7fc06be00640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 0x7fc06c1a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.911+0000 7fc071400640 1 -- 172.21.15.123:0/3089188168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 msgr2=0x7fc06c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.911+0000 7fc071400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c1a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.911+0000 7fc071400640 1 -- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 msgr2=0x7fc06c1a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.911+0000 7fc071400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 0x7fc06c1a6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.911+0000 7fc071400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c1ae410 con 0x7fc06c10dd70 2024-03-24T00:44:45.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.912+0000 7fc071400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c1adda0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc064005fd0 tx=0x7fc06400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.912+0000 7fc069600640 1 -- 172.21.15.123:0/3089188168 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06400cdf0 con 0x7fc06c10dd70 2024-03-24T00:44:45.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.913+0000 7fc069600640 1 -- 172.21.15.123:0/3089188168 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc064005020 con 0x7fc06c10dd70 2024-03-24T00:44:45.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.913+0000 7fc069600640 1 -- 172.21.15.123:0/3089188168 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640106d0 con 0x7fc06c10dd70 2024-03-24T00:44:45.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.913+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c1ae700 con 0x7fc06c10dd70 2024-03-24T00:44:45.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.913+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c1aec00 con 0x7fc06c10dd70 2024-03-24T00:44:45.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.920+0000 7fc069600640 1 -- 172.21.15.123:0/3089188168 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc0640054d0 con 0x7fc06c10dd70 2024-03-24T00:44:45.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.924+0000 7fc069600640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 0x7fc0540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.925+0000 7fc069600640 1 -- 172.21.15.123:0/3089188168 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc0640cf0d0 con 0x7fc06c10dd70 2024-03-24T00:44:45.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.926+0000 7fc070a00640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 0x7fc0540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.927+0000 7fc070a00640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 0x7fc0540a4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc06000d070 tx=0x7fc06000d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.934+0000 7fc07607d740 1 Processor -- start 2024-03-24T00:44:45.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.934+0000 7fc07607d740 1 -- start start 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a601250 con 0x55f83a69c870 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a5a9a30 con 0x55f83a6a19e0 2024-03-24T00:44:45.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc07607d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a5fd2f0 con 0x55f83a69be80 2024-03-24T00:44:45.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc070a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc06be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc070a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49894/0 (socket says 172.21.15.123:49894) 2024-03-24T00:44:45.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.936+0000 7fc070a00640 1 -- 172.21.15.123:0/909437997 learned_addr learned my addr 172.21.15.123:0/909437997 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:45.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.937+0000 7fc071400640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.938+0000 7fc070a00640 1 -- 172.21.15.123:0/909437997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 msgr2=0x55f83a6a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:45.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.938+0000 7fc070a00640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.938+0000 7fc070a00640 1 -- 172.21.15.123:0/909437997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 msgr2=0x55f83a6a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:45.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.938+0000 7fc070a00640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.938+0000 7fc070a00640 1 -- 172.21.15.123:0/909437997 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f83a594690 con 0x55f83a69be80 2024-03-24T00:44:45.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.939+0000 7fc071400640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:45.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.939+0000 7fc06be00640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:45.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.939+0000 7fc070a00640 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc06002cec0 tx=0x7fc06003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d07999b67cdaaf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.940+0000 7fc05dc00640 1 -- 172.21.15.123:0/909437997 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06004d030 con 0x55f83a69be80 2024-03-24T00:44:45.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.940+0000 7fc05dc00640 1 -- 172.21.15.123:0/909437997 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc060004ba0 con 0x55f83a69be80 2024-03-24T00:44:45.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.940+0000 7fc05dc00640 1 -- 172.21.15.123:0/909437997 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060008260 con 0x55f83a69be80 2024-03-24T00:44:45.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.942+0000 7fc07607d740 1 -- 172.21.15.123:0/909437997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 msgr2=0x55f83a69c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.942+0000 7fc07607d740 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc06002cec0 tx=0x7fc06003e000 comp rx=0 tx=0).stop 2024-03-24T00:44:45.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.943+0000 7fc07607d740 1 -- 172.21.15.123:0/909437997 shutdown_connections 2024-03-24T00:44:45.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.943+0000 7fc07607d740 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.943+0000 7fc07607d740 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.943+0000 7fc07607d740 1 --2- 172.21.15.123:0/909437997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a69c2a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.944+0000 7fc07607d740 1 -- 172.21.15.123:0/909437997 >> 172.21.15.123:0/909437997 conn(0x55f83a664200 msgr2=0x55f83a6977f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:45.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.944+0000 7fc07607d740 1 -- 172.21.15.123:0/909437997 shutdown_connections 2024-03-24T00:44:45.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.944+0000 7fc07607d740 1 -- 172.21.15.123:0/909437997 wait complete. 2024-03-24T00:44:45.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.945+0000 7fc07607d740 1 -- start start 2024-03-24T00:44:45.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.945+0000 7fc07607d740 1 Processor -- start 2024-03-24T00:44:45.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc070a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6ad380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a6a4930 con 0x55f83a69c870 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a5bce80 con 0x55f83a6a19e0 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.947+0000 7fc07607d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83a6a4ab0 con 0x55f83a69be80 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.948+0000 7fc07607d740 10 client.0 _ll_register_callbacks cb 0x55f83a5cf130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.948+0000 7fc070a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49910/0 (socket says 172.21.15.123:49910) 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.948+0000 7fc070a00640 1 -- 172.21.15.123:0/416208456 learned_addr learned my addr 172.21.15.123:0/416208456 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.949+0000 7fc070a00640 1 -- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 msgr2=0x55f83a6ad380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:44:45.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.949+0000 7fc070a00640 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6ad380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.949+0000 7fc070a00640 1 -- 172.21.15.123:0/416208456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 msgr2=0x55f83a6a6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.949+0000 7fc070a00640 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.949+0000 7fc070a00640 1 -- 172.21.15.123:0/416208456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f83a3df7d0 con 0x55f83a69be80 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.950+0000 7fc070a00640 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc060005ec0 tx=0x7fc060004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8465a2ba47d0cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.950+0000 7fc052000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:45.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.950+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06004d040 con 0x55f83a69be80 2024-03-24T00:44:45.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.950+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc060005180 con 0x55f83a69be80 2024-03-24T00:44:45.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.951+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc060004ba0 con 0x55f83a69be80 2024-03-24T00:44:45.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.951+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc060005380 con 0x55f83a69be80 2024-03-24T00:44:45.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.951+0000 7fc07607d740 10 client.26332 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:45.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f83a3153f0 con 0x55f83a69be80 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc060004ba0 con 0x55f83a69be80 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc060058020 con 0x55f83a69be80 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 10 client.26332 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 10 client.26332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 20 client.26332 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:45.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a6b1870 0x55f83a6b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 4 client.26332 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.952+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f83a60e790 con 0x55f83a6b1870 2024-03-24T00:44:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.953+0000 7fc06be00640 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a6b1870 0x55f83a6b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.955+0000 7fc06be00640 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a6b1870 0x55f83a6b3c60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7d739f4a6e19787 server_cookie=eceb25755027b99f in_seq=0 out_seq=0 2024-03-24T00:44:45.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:45.955+0000 7fc052000640 10 client.26332 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:46 smithi177 ceph-mon[36034]: pgmap v591: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 41 MiB/s wr, 3.91k op/s 2024-03-24T00:44:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:46 smithi204 ceph-mon[36345]: pgmap v591: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 41 MiB/s wr, 3.91k op/s 2024-03-24T00:44:46.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:46 smithi123 ceph-mon[31487]: pgmap v591: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 41 MiB/s wr, 3.91k op/s 2024-03-24T00:44:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:48 smithi177 ceph-mon[36034]: pgmap v592: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 40 MiB/s wr, 3.57k op/s 2024-03-24T00:44:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:48 smithi204 ceph-mon[36345]: pgmap v592: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 40 MiB/s wr, 3.57k op/s 2024-03-24T00:44:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:48 smithi123 ceph-mon[31487]: pgmap v592: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 40 MiB/s wr, 3.57k op/s 2024-03-24T00:44:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:50 smithi177 ceph-mon[36034]: pgmap v593: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 3.77k op/s 2024-03-24T00:44:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:50 smithi204 ceph-mon[36345]: pgmap v593: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 3.77k op/s 2024-03-24T00:44:50.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:50 smithi123 ceph-mon[31487]: pgmap v593: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 3.77k op/s 2024-03-24T00:44:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc257e00640 1 -- 172.21.15.123:0/640155106 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d59289a0c0 con 0x55d592932f50 2024-03-24T00:44:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc257e00640 10 client.26312 handle_command_reply: tid=0 2024-03-24T00:44:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 client.26312 shutdown 2024-03-24T00:44:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 msgr2=0x55d59293d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d59293d640 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc26c09b890 tx=0x7fc26c097eb0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc052000640 1 -- 172.21.15.123:0/416208456 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f83a60e790 con 0x55f83a6b1870 2024-03-24T00:44:51.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc052000640 10 client.26332 handle_command_reply: tid=0 2024-03-24T00:44:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 shutdown_connections 2024-03-24T00:44:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d59292dde0 0x55d59293d640 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d59292d3f0 0x55d59293c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d592932f50 0x55d5929a16b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.197+0000 7fc07607d740 1 client.26332 shutdown 2024-03-24T00:44:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc27c3e7740 1 --2- 172.21.15.123:0/640155106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d592928960 0x55d59293c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc257e00640 0 client.26312 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 >> 172.21.15.123:0/640155106 conn(0x55d592932a50 msgr2=0x55d59292f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 shutdown_connections 2024-03-24T00:44:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc27c3e7740 1 -- 172.21.15.123:0/640155106 wait complete. 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 msgr2=0x55f83a6a5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc060005ec0 tx=0x7fc060004a90 comp rx=0 tx=0).stop 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 shutdown_connections 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a6a19e0 0x55f83a6ad380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69c870 0x55f83a6a6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a6b1870 0x55f83a6b3c60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 --2- 172.21.15.123:0/416208456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a69be80 0x55f83a6a5e00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc052000640 0 client.26332 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 >> 172.21.15.123:0/416208456 conn(0x55f83a664200 msgr2=0x55f83a696a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc27c3e7740 20 client.26312 trim_cache size 0 max 16384 2024-03-24T00:44:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 shutdown_connections 2024-03-24T00:44:51.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 1 -- 172.21.15.123:0/416208456 wait complete. 2024-03-24T00:44:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:44:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:44:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:44:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:44:51.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.198+0000 7fc07607d740 20 client.26332 trim_cache size 0 max 16384 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 msgr2=0x7fc2580a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 0x7fc2580a4cb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc25c02c170 tx=0x7fc25c02c8a0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 msgr2=0x7fc2741a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 0x7fc2741a0fd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009fd0 tx=0x7fc26c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 shutdown_connections 2024-03-24T00:44:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2741071d0 0x7fc2741a0fd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc2580a27f0 0x7fc2580a4cb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2741048a0 0x7fc274199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 --2- 172.21.15.123:0/736688783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc274101f70 0x7fc274199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 >> 172.21.15.123:0/736688783 conn(0x7fc2740fb980 msgr2=0x7fc274101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 shutdown_connections 2024-03-24T00:44:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:44:51.199+0000 7fc273e00640 1 -- 172.21.15.123:0/736688783 wait complete. 2024-03-24T00:44:51.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 Processor -- start 2024-03-24T00:44:51.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 -- start start 2024-03-24T00:44:51.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a69dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a696910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b59a0 con 0x55f83a69be80 2024-03-24T00:44:51.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b7f60 con 0x55f83a69c870 2024-03-24T00:44:51.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.221+0000 7fc07607d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b5fa0 con 0x55f83a6a19e0 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc070a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a69dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc071400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a696910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49996/0 (socket says 172.21.15.123:49996) 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 -- 172.21.15.123:0/4262683467 learned_addr learned my addr 172.21.15.123:0/4262683467 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 -- 172.21.15.123:0/4262683467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 msgr2=0x55f83a696910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a696910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 -- 172.21.15.123:0/4262683467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 msgr2=0x55f83a69dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a69dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc06be00640 1 -- 172.21.15.123:0/4262683467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f83a2d50f0 con 0x55f83a69c870 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc070a00640 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a69dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.222+0000 7fc071400640 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a696910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.223+0000 7fc06be00640 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc058009730 tx=0x7fc058009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1615f8c9a0a95f0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.223+0000 7fc05d200640 1 -- 172.21.15.123:0/4262683467 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058020030 con 0x55f83a69c870 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.223+0000 7fc05d200640 1 -- 172.21.15.123:0/4262683467 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc058005500 con 0x55f83a69c870 2024-03-24T00:44:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.223+0000 7fc05d200640 1 -- 172.21.15.123:0/4262683467 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05801c870 con 0x55f83a69c870 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 -- 172.21.15.123:0/4262683467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 msgr2=0x55f83a6ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc058009730 tx=0x7fc058009f90 comp rx=0 tx=0).stop 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 -- 172.21.15.123:0/4262683467 shutdown_connections 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a696910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ad310 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 --2- 172.21.15.123:0/4262683467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a69dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.224+0000 7fc07607d740 1 -- 172.21.15.123:0/4262683467 >> 172.21.15.123:0/4262683467 conn(0x55f83a664200 msgr2=0x55f83a696200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.225+0000 7fc07607d740 1 -- 172.21.15.123:0/4262683467 shutdown_connections 2024-03-24T00:44:51.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.225+0000 7fc07607d740 1 -- 172.21.15.123:0/4262683467 wait complete. 2024-03-24T00:44:51.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.225+0000 7fc07607d740 1 -- start start 2024-03-24T00:44:51.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.225+0000 7fc07607d740 1 Processor -- start 2024-03-24T00:44:51.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a6ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a6ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b65e0 con 0x55f83a69be80 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b7900 con 0x55f83a69c870 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1b6c40 con 0x55f83a6a19e0 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc07607d740 10 client.0 _ll_register_callbacks cb 0x7fc0580097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc06be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc06be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49998/0 (socket says 172.21.15.123:49998) 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc06be00640 1 -- 172.21.15.123:0/3072431542 learned_addr learned my addr 172.21.15.123:0/3072431542 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc070a00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a6ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.226+0000 7fc071400640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a6ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 -- 172.21.15.123:0/3072431542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 msgr2=0x55f83a6ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a6ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 -- 172.21.15.123:0/3072431542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 msgr2=0x55f83a6ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a6ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 -- 172.21.15.123:0/3072431542 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f83a6ace20 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc06be00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc058009e60 tx=0x7fc058009e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73c3a952966ed65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc052000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058020070 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.227+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc058004700 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc058005500 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc058004900 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 10 client.26327 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f83a70d8a0 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc058005500 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc058025500 con 0x55f83a69c870 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 10 client.26327 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 10 client.26327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:51.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 20 client.26327 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a70e210 0x55f83a710630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 4 client.26327 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.228+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f83a608ae0 con 0x55f83a70e210 2024-03-24T00:44:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.229+0000 7fc070a00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a70e210 0x55f83a710630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.229+0000 7fc052000640 10 client.26327 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.229+0000 7fc070a00640 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a70e210 0x55f83a710630 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b30633e3ef7d96 server_cookie=26dc316768eee994 in_seq=0 out_seq=0 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.229+0000 7fc052000640 1 -- 172.21.15.123:0/3072431542 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f83a608ae0 con 0x55f83a70e210 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.229+0000 7fc052000640 10 client.26327 handle_command_reply: tid=0 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 client.26327 shutdown 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 msgr2=0x55f83a6ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc058009e60 tx=0x7fc058009e90 comp rx=0 tx=0).stop 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 shutdown_connections 2024-03-24T00:44:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f83a70e210 0x55f83a710630 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f83a6a19e0 0x55f83a6ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a69c870 0x55f83a6ab820 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc052000640 0 client.26327 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 --2- 172.21.15.123:0/3072431542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f83a69be80 0x55f83a6ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 >> 172.21.15.123:0/3072431542 conn(0x55f83a664200 msgr2=0x55f83a69d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 shutdown_connections 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 1 -- 172.21.15.123:0/3072431542 wait complete. 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.230+0000 7fc07607d740 20 client.26327 trim_cache size 0 max 16384 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:44:51.234 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 msgr2=0x7fc0540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 0x7fc0540a4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc06000d070 tx=0x7fc06000d000 comp rx=0 tx=0).stop 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 msgr2=0x7fc06c1adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c1adda0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc064005fd0 tx=0x7fc06400ef10 comp rx=0 tx=0).stop 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 shutdown_connections 2024-03-24T00:44:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc06c10dd70 0x7fc06c1adda0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc0540a2880 0x7fc0540a4d40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc06c10b2d0 0x7fc06c1a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 --2- 172.21.15.123:0/3089188168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc06c108910 0x7fc06c1a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 >> 172.21.15.123:0/3089188168 conn(0x7fc06c06e500 msgr2=0x7fc06c109f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 shutdown_connections 2024-03-24T00:44:51.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.231+0000 7fc073400640 1 -- 172.21.15.123:0/3089188168 wait complete. 2024-03-24T00:44:51.251 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:44:51.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 -- 172.21.15.123:0/1647165090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88109390 msgr2=0x7fee8810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 --2- 172.21.15.123:0/1647165090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88109390 0x7fee8810b820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fee80009ca0 tx=0x7fee8002c180 comp rx=0 tx=0).stop 2024-03-24T00:44:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 -- 172.21.15.123:0/1647165090 shutdown_connections 2024-03-24T00:44:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 --2- 172.21.15.123:0/1647165090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88109390 0x7fee8810b820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 --2- 172.21.15.123:0/1647165090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88074080 0x7fee880744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 --2- 172.21.15.123:0/1647165090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88075660 0x7fee88073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 -- 172.21.15.123:0/1647165090 >> 172.21.15.123:0/1647165090 conn(0x7fee880fca30 msgr2=0x7fee880fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 -- 172.21.15.123:0/1647165090 shutdown_connections 2024-03-24T00:44:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.444+0000 7fee87200640 1 -- 172.21.15.123:0/1647165090 wait complete. 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.445+0000 7fee87200640 1 Processor -- start 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.445+0000 7fee87200640 1 -- start start 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88075660 0x7fee88199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88109390 0x7fee881a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8810f370 con 0x7fee88075660 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8810f1f0 con 0x7fee88109390 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee87200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8810f4f0 con 0x7fee88074080 2024-03-24T00:44:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49956/0 (socket says 172.21.15.123:49956) 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 -- 172.21.15.123:0/2729548226 learned_addr learned my addr 172.21.15.123:0/2729548226 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85400640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88075660 0x7fee88199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee86800640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88109390 0x7fee881a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 -- 172.21.15.123:0/2729548226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88109390 msgr2=0x7fee881a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88109390 0x7fee881a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 -- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88075660 msgr2=0x7fee88199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88075660 0x7fee88199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.446+0000 7fee85e00640 1 -- 172.21.15.123:0/2729548226 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee881a15c0 con 0x7fee88074080 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee85e00640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fee7800e9f0 tx=0x7fee7800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7800cd80 con 0x7fee88074080 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee78005500 con 0x7fee88074080 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee881a18b0 con 0x7fee88074080 2024-03-24T00:44:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78010690 con 0x7fee88074080 2024-03-24T00:44:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.447+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee881a1db0 con 0x7fee88074080 2024-03-24T00:44:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.448+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee88108150 con 0x7fee88074080 2024-03-24T00:44:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.450+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee780046c0 con 0x7fee88074080 2024-03-24T00:44:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.452+0000 7fee7ea00640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 0x7fee6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.452+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fee780ceea0 con 0x7fee88074080 2024-03-24T00:44:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.452+0000 7fee85400640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 0x7fee6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.453+0000 7fee85400640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 0x7fee6c0a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fee740095f0 tx=0x7fee74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.457+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee7809bef0 con 0x7fee88074080 2024-03-24T00:44:51.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.710+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee8802d070 con 0x7fee88074080 2024-03-24T00:44:51.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.711+0000 7fee7ea00640 1 -- 172.21.15.123:0/2729548226 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fee7809b7f0 con 0x7fee88074080 2024-03-24T00:44:51.713 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:44:51.713 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:44:51.713 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 msgr2=0x7fee6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 0x7fee6c0a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fee740095f0 tx=0x7fee74005f90 comp rx=0 tx=0).stop 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 msgr2=0x7fee881999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fee7800e9f0 tx=0x7fee7800edf0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 shutdown_connections 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee88109390 0x7fee881a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fee6c0a2800 0x7fee6c0a4cc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee88075660 0x7fee88199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 --2- 172.21.15.123:0/2729548226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee88074080 0x7fee881999a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 >> 172.21.15.123:0/2729548226 conn(0x7fee880fca30 msgr2=0x7fee88105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 shutdown_connections 2024-03-24T00:44:51.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.714+0000 7fee87200640 1 -- 172.21.15.123:0/2729548226 wait complete. 2024-03-24T00:44:51.733 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:44:51.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:44:51.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 -- 172.21.15.123:0/1677824110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44075660 msgr2=0x7f7b44073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1677824110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44075660 0x7f7b44073ab0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009900 tx=0x7f7b3402bcd0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 -- 172.21.15.123:0/1677824110 shutdown_connections 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1677824110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b4410b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1677824110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44074080 0x7f7b440744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1677824110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44075660 0x7f7b44073ab0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 -- 172.21.15.123:0/1677824110 >> 172.21.15.123:0/1677824110 conn(0x7f7b440fcb10 msgr2=0x7f7b440fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 -- 172.21.15.123:0/1677824110 shutdown_connections 2024-03-24T00:44:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.930+0000 7f7b43e00640 1 -- 172.21.15.123:0/1677824110 wait complete. 2024-03-24T00:44:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.931+0000 7f7b43e00640 1 Processor -- start 2024-03-24T00:44:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.931+0000 7f7b43e00640 1 -- start start 2024-03-24T00:44:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 0x7f7b44199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b441a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4410f420 con 0x7f7b44075660 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4410f2a0 con 0x7f7b44074080 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4410f5a0 con 0x7f7b44109450 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50022/0 (socket says 172.21.15.123:50022) 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 -- 172.21.15.123:0/1909252636 learned_addr learned my addr 172.21.15.123:0/1909252636 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42000640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 0x7f7b44199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43400640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b441a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 -- 172.21.15.123:0/1909252636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 msgr2=0x7f7b441a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b441a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 -- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 msgr2=0x7f7b44199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 0x7f7b44199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42a00640 1 -- 172.21.15.123:0/1909252636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b441a1690 con 0x7f7b44074080 2024-03-24T00:44:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b43400640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b441a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.932+0000 7f7b42000640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 0x7f7b44199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b42a00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7b34005ab0 tx=0x7f7b3402c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b3b400640 1 -- 172.21.15.123:0/1909252636 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34041070 con 0x7f7b44074080 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b3b400640 1 -- 172.21.15.123:0/1909252636 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b34004450 con 0x7f7b44074080 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b441a1920 con 0x7f7b44074080 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b3b400640 1 -- 172.21.15.123:0/1909252636 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34034440 con 0x7f7b44074080 2024-03-24T00:44:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.933+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441a1e10 con 0x7f7b44074080 2024-03-24T00:44:51.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.936+0000 7f7b3b400640 1 -- 172.21.15.123:0/1909252636 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7b34044070 con 0x7f7b44074080 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.936+0000 7f7b4c7cc740 1 Processor -- start 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 -- start start 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d35917040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d3591de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3587a2c0 con 0x556d35916bc0 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d35822c90 con 0x556d35917580 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d35876360 con 0x556d359159f0 2024-03-24T00:44:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b43400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b43400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49972/0 (socket says 172.21.15.123:49972) 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b43400640 1 -- 172.21.15.123:0/3076109966 learned_addr learned my addr 172.21.15.123:0/3076109966 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b42000640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d3591de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.937+0000 7f7b42a00640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d35917040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 -- 172.21.15.123:0/3076109966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 msgr2=0x556d3591de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d3591de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 -- 172.21.15.123:0/3076109966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 msgr2=0x556d35917040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d35917040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 -- 172.21.15.123:0/3076109966 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d35847080 con 0x556d359159f0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b42a00640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d35917040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b3b400640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 0x7f7b280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b3b400640 1 -- 172.21.15.123:0/1909252636 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7b340f0540 con 0x7f7b44074080 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b43400640 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0117e0 tx=0x7f7b3c011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a061e87c79a2f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b42a00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 0x7f7b280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b2fe00640 1 -- 172.21.15.123:0/3076109966 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c011310 con 0x556d359159f0 2024-03-24T00:44:51.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b2fe00640 1 -- 172.21.15.123:0/3076109966 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c003e50 con 0x556d359159f0 2024-03-24T00:44:51.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.938+0000 7f7b2fe00640 1 -- 172.21.15.123:0/3076109966 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c0045b0 con 0x556d359159f0 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3076109966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 msgr2=0x556d35915e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0117e0 tx=0x7f7b3c011ca0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b42a00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 0x7f7b280a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009430 tx=0x7f7b3402bcd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3076109966 shutdown_connections 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d3591de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d35917040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3076109966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35915e10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3076109966 >> 172.21.15.123:0/3076109966 conn(0x556d35910de0 msgr2=0x556d359111e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3076109966 shutdown_connections 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3076109966 wait complete. 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.939+0000 7f7b4c7cc740 1 -- start start 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 Processor -- start 2024-03-24T00:44:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d359159f0 0x556d3591fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d359271e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3591e590 con 0x556d359159f0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d358360d0 con 0x556d35917580 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3591e710 con 0x556d35916bc0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b4c7cc740 10 client.0 _ll_register_callbacks cb 0x556d3583ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b42000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b42000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:49976/0 (socket says 172.21.15.123:49976) 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b42000640 1 -- 172.21.15.123:0/3602501425 learned_addr learned my addr 172.21.15.123:0/3602501425 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.940+0000 7f7b43400640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d359159f0 0x556d3591fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b43400640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d359271e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 -- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 msgr2=0x556d359271e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d359271e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 -- 172.21.15.123:0/3602501425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d359159f0 msgr2=0x556d3591fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d359159f0 0x556d3591fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 -- 172.21.15.123:0/3602501425 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d3558e3f0 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b43400640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d359271e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b42000640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b30012d00 tx=0x7f7b3000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=568fa2348587e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.941+0000 7f7b2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30012110 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7b30004050 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b3c003e50 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b300122b0 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b4c7cc740 10 client.26356 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556d356587d0 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c003e50 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.942+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7b30010430 con 0x556d35916bc0 2024-03-24T00:44:51.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 10 client.26356 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 10 client.26356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 20 client.26356 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d3592b7a0 0x556d3592dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 4 client.26356 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d35887800 con 0x556d3592b7a0 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b43400640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d3592b7a0 0x556d3592dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b43400640 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d3592b7a0 0x556d3592dc00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=869a6344696f14ea server_cookie=40e87fe93caa7d87 in_seq=0 out_seq=0 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.943+0000 7f7b2cc00640 10 client.26356 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b2cc00640 1 -- 172.21.15.123:0/3602501425 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d35887800 con 0x556d3592b7a0 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b2cc00640 10 client.26356 handle_command_reply: tid=0 2024-03-24T00:44:51.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 client.26356 shutdown 2024-03-24T00:44:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 msgr2=0x556d359201a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b30012d00 tx=0x7f7b3000f790 comp rx=0 tx=0).stop 2024-03-24T00:44:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 shutdown_connections 2024-03-24T00:44:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35917580 0x556d359271e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d3592b7a0 0x556d3592dc00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35916bc0 0x556d359201a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/3602501425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d359159f0 0x556d3591fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b2cc00640 0 client.26356 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.944+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 >> 172.21.15.123:0/3602501425 conn(0x556d35910de0 msgr2=0x556d35910160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.945+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 shutdown_connections 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.945+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/3602501425 wait complete. 2024-03-24T00:44:51.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.945+0000 7f7b4c7cc740 20 client.26356 trim_cache size 0 max 16384 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 Processor -- start 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 -- start start 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d359271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35917580 0x556d35919c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a8b00 con 0x556d35916bc0 2024-03-24T00:44:51.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441ab0c0 con 0x556d359159f0 2024-03-24T00:44:51.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.960+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a9100 con 0x556d35917580 2024-03-24T00:44:51.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b42000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d359271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50054/0 (socket says 172.21.15.123:50054) 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 -- 172.21.15.123:0/1456800008 learned_addr learned my addr 172.21.15.123:0/1456800008 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35917580 0x556d35919c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 -- 172.21.15.123:0/1456800008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35917580 msgr2=0x556d35919c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35917580 0x556d35919c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 -- 172.21.15.123:0/1456800008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 msgr2=0x556d359271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d359271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 -- 172.21.15.123:0/1456800008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d3592b580 con 0x556d359159f0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b43400640 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009560 tx=0x7f7b3c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b69a955fe335939 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.961+0000 7f7b2f400640 1 -- 172.21.15.123:0/1456800008 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c022040 con 0x556d359159f0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b2f400640 1 -- 172.21.15.123:0/1456800008 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c00fce0 con 0x556d359159f0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b2f400640 1 -- 172.21.15.123:0/1456800008 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c02ae60 con 0x556d359159f0 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1456800008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 msgr2=0x556d35914750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009560 tx=0x7f7b3c005bd0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1456800008 shutdown_connections 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d35917580 0x556d35919c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35916bc0 0x556d359271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1456800008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d359159f0 0x556d35914750 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1456800008 >> 172.21.15.123:0/1456800008 conn(0x556d35910de0 msgr2=0x556d35914040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1456800008 shutdown_connections 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.962+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1456800008 wait complete. 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 -- start start 2024-03-24T00:44:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 Processor -- start 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35923060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 0x556d359265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a9740 con 0x556d35917580 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441aaa60 con 0x556d35916bc0 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a9da0 con 0x556d359159f0 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b4c7cc740 10 client.0 _ll_register_callbacks cb 0x7f7b30009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.963+0000 7f7b42000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50070/0 (socket says 172.21.15.123:50070) 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 -- 172.21.15.123:0/1854881301 learned_addr learned my addr 172.21.15.123:0/1854881301 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b43400640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35923060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b43400640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 0x556d359265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 -- 172.21.15.123:0/1854881301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 msgr2=0x556d35923060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35923060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 -- 172.21.15.123:0/1854881301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 msgr2=0x556d359265c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 0x556d359265c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 -- 172.21.15.123:0/1854881301 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d35920d90 con 0x556d35916bc0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b43400640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 0x556d359265c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b42000640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000a010 tx=0x7f7b30012110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c54b23ce7b7743 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.964+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30005610 con 0x556d35916bc0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7b30010a90 con 0x556d35916bc0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b3c00fce0 con 0x556d35916bc0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b3000fe80 con 0x556d35916bc0 2024-03-24T00:44:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 10 client.26348 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556d35987640 con 0x556d35916bc0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c00fce0 con 0x556d35916bc0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7b30027450 con 0x556d35916bc0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 10 client.26348 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 10 client.26348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 20 client.26348 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d35987f10 0x556d3598a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 4 client.26348 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.965+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d35881b50 con 0x556d35987f10 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b43400640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d35987f10 0x556d3598a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b43400640 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d35987f10 0x556d3598a330 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0d53ef77ce040be server_cookie=fd0ed6293de4fa35 in_seq=0 out_seq=0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b2cc00640 10 client.26348 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b2cc00640 1 -- 172.21.15.123:0/1854881301 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d35881b50 con 0x556d35987f10 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b2cc00640 10 client.26348 handle_command_reply: tid=0 2024-03-24T00:44:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 client.26348 shutdown 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 msgr2=0x556d359235a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000a010 tx=0x7f7b30012110 comp rx=0 tx=0).stop 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 shutdown_connections 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556d35917580 0x556d359265c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d35916bc0 0x556d359235a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556d35987f10 0x556d3598a330 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b4c7cc740 1 --2- 172.21.15.123:0/1854881301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556d359159f0 0x556d35923060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.966+0000 7f7b2cc00640 0 client.26348 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 >> 172.21.15.123:0/1854881301 conn(0x556d35910de0 msgr2=0x556d359148d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 shutdown_connections 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b4c7cc740 1 -- 172.21.15.123:0/1854881301 wait complete. 2024-03-24T00:44:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b4c7cc740 20 client.26348 trim_cache size 0 max 16384 2024-03-24T00:44:51.973 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:44:51.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 msgr2=0x7f7b280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 0x7f7b280a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009430 tx=0x7f7b3402bcd0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 msgr2=0x7f7b44199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7b34005ab0 tx=0x7f7b3402c6e0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 shutdown_connections 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b44109450 0x7f7b441a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7b280a2800 0x7f7b280a4cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b44075660 0x7f7b44199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 --2- 172.21.15.123:0/1909252636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b44074080 0x7f7b44199a70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.967+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 >> 172.21.15.123:0/1909252636 conn(0x7f7b440fcb10 msgr2=0x7f7b441053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.968+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 shutdown_connections 2024-03-24T00:44:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:51.968+0000 7f7b43e00640 1 -- 172.21.15.123:0/1909252636 wait complete. 2024-03-24T00:44:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:52 smithi204 ceph-mon[36345]: pgmap v594: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 46 MiB/s wr, 4.17k op/s 2024-03-24T00:44:52.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:52 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2729548226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:52 smithi123 ceph-mon[31487]: pgmap v594: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 46 MiB/s wr, 4.17k op/s 2024-03-24T00:44:52.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:52 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2729548226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:52 smithi177 ceph-mon[36034]: pgmap v594: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 46 MiB/s wr, 4.17k op/s 2024-03-24T00:44:52.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:52 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2729548226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:44:52.985 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:44:52.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:52.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 -- 172.21.15.123:0/1631341754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0104930 msgr2=0x7f38c0106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 --2- 172.21.15.123:0/1631341754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0104930 0x7f38c0106d20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009900 tx=0x7f38b002bc30 comp rx=0 tx=0).stop 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 -- 172.21.15.123:0/1631341754 shutdown_connections 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 --2- 172.21.15.123:0/1631341754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0107260 0x7f38c01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 --2- 172.21.15.123:0/1631341754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0104930 0x7f38c0106d20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 --2- 172.21.15.123:0/1631341754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 -- 172.21.15.123:0/1631341754 >> 172.21.15.123:0/1631341754 conn(0x7f38c00fba10 msgr2=0x7f38c00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 -- 172.21.15.123:0/1631341754 shutdown_connections 2024-03-24T00:44:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.192+0000 7f38c7200640 1 -- 172.21.15.123:0/1631341754 wait complete. 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 Processor -- start 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 -- start start 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 0x7f38c0197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0107260 0x7f38c019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010d260 con 0x7f38c0107260 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010d0e0 con 0x7f38c0102000 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010d3e0 con 0x7f38c0104930 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.193+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 0x7f38c0197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50100/0 (socket says 172.21.15.123:50100) 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 -- 172.21.15.123:0/756222155 learned_addr learned my addr 172.21.15.123:0/756222155 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c6800640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0107260 0x7f38c019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 -- 172.21.15.123:0/756222155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 msgr2=0x7f38c0197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 0x7f38c0197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 -- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0107260 msgr2=0x7f38c019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0107260 0x7f38c019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 -- 172.21.15.123:0/756222155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c019f4d0 con 0x7f38c0102000 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5400640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 0x7f38c0197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38c5e00640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f38b400ea60 tx=0x7f38b400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.194+0000 7f38baa00640 1 -- 172.21.15.123:0/756222155 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b400cde0 con 0x7f38c0102000 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.195+0000 7f38baa00640 1 -- 172.21.15.123:0/756222155 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38b40035f0 con 0x7f38c0102000 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.195+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c019f7c0 con 0x7f38c0102000 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.195+0000 7f38baa00640 1 -- 172.21.15.123:0/756222155 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4010670 con 0x7f38c0102000 2024-03-24T00:44:53.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.195+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c019fd30 con 0x7f38c0102000 2024-03-24T00:44:53.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38baa00640 1 -- 172.21.15.123:0/756222155 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38b40037a0 con 0x7f38c0102000 2024-03-24T00:44:53.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 Processor -- start 2024-03-24T00:44:53.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 -- start start 2024-03-24T00:44:53.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 0x561f6577f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f657847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f657872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f656e4eb0 con 0x561f6577f2b0 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6568da60 con 0x561f65784e10 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f656e0f50 con 0x561f6577fca0 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f657847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38c6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 0x561f6577f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 0x561f6577f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46988/0 (socket says 172.21.15.123:46988) 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f657847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50014/0 (socket says 172.21.15.123:50014) 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c6800640 1 -- 172.21.15.123:0/740339895 learned_addr learned my addr 172.21.15.123:0/740339895 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.198+0000 7f38c5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f657872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 -- 172.21.15.123:0/740339895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 msgr2=0x561f657847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f657847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 -- 172.21.15.123:0/740339895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 msgr2=0x561f6577f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 0x561f6577f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 -- 172.21.15.123:0/740339895 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f654c17e0 con 0x561f65784e10 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38c5400640 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f657872a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009230 tx=0x7f38b002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62ed90aa29b47cdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38baa00640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 0x7f38a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.199+0000 7f38af000640 1 -- 172.21.15.123:0/740339895 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0045070 con 0x561f65784e10 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38af000640 1 -- 172.21.15.123:0/740339895 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38b002cd30 con 0x561f65784e10 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38af000640 1 -- 172.21.15.123:0/740339895 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00050d0 con 0x561f65784e10 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38baa00640 1 -- 172.21.15.123:0/756222155 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f38b40ced20 con 0x7f38c0102000 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38c6800640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 0x7f38a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 -- 172.21.15.123:0/740339895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 msgr2=0x561f657872a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f657872a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009230 tx=0x7f38b002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38c6800640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 0x7f38a80a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x561f65780500 tx=0x7f38bc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 -- 172.21.15.123:0/740339895 shutdown_connections 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f657872a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f657847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/740339895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577f2b0 0x561f6577f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.200+0000 7f38cb5aa740 1 -- 172.21.15.123:0/740339895 >> 172.21.15.123:0/740339895 conn(0x561f6577a820 msgr2=0x561f6577ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.201+0000 7f38cb5aa740 1 -- 172.21.15.123:0/740339895 shutdown_connections 2024-03-24T00:44:53.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.201+0000 7f38cb5aa740 1 -- 172.21.15.123:0/740339895 wait complete. 2024-03-24T00:44:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.201+0000 7f38cb5aa740 1 -- start start 2024-03-24T00:44:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.201+0000 7f38cb5aa740 1 Processor -- start 2024-03-24T00:44:53.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 0x561f6578dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f65794d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6578c170 con 0x561f6577fca0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f656a0eb0 con 0x561f65784e10 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6578c2f0 con 0x561f6577f2b0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38cb5aa740 10 client.0 _ll_register_callbacks cb 0x561f65695150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 0x561f6578dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50022/0 (socket says 172.21.15.123:50022) 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 -- 172.21.15.123:0/728552832 learned_addr learned my addr 172.21.15.123:0/728552832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c6800640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f65794d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 -- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 msgr2=0x561f65794d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f65794d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 -- 172.21.15.123:0/728552832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 msgr2=0x561f6578dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 0x561f6578dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5e00640 1 -- 172.21.15.123:0/728552832 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f65521f70 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c6800640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f65794d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.202+0000 7f38c5400640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 0x561f6578dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38c5e00640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38b40a38e0 tx=0x7f38b40979d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a9767f5a9fc6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40bb050 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f38b4003b30 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38b002cd30 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38b409afd0 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 10 client.26374 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561f653b9170 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38b002cd30 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f38b4097ce0 con 0x561f6577f2b0 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 10 client.26374 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 10 client.26374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:53.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 20 client.26374 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.203+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657995c0 0x561f6579ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.204+0000 7f38cb5aa740 4 client.26374 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.204+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f656f23f0 con 0x561f657995c0 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.204+0000 7f38c5400640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657995c0 0x561f6579ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.204+0000 7f38a3400640 10 client.26374 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.204+0000 7f38c5400640 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657995c0 0x561f6579ba30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a242bd940e503c69 server_cookie=f8b86ec8c604b25d in_seq=0 out_seq=0 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38a3400640 1 -- 172.21.15.123:0/728552832 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561f656f23f0 con 0x561f657995c0 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38a3400640 10 client.26374 handle_command_reply: tid=0 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 client.26374 shutdown 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 msgr2=0x561f6578d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38b40a38e0 tx=0x7f38b40979d0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 shutdown_connections 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f65784e10 0x561f65794d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f6577fca0 0x561f6578dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657995c0 0x561f6579ba30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/728552832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f6578d7c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38a3400640 0 client.26374 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 >> 172.21.15.123:0/728552832 conn(0x561f6577a820 msgr2=0x561f65779a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.206+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 shutdown_connections 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.207+0000 7f38cb5aa740 1 -- 172.21.15.123:0/728552832 wait complete. 2024-03-24T00:44:53.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.207+0000 7f38cb5aa740 20 client.26374 trim_cache size 0 max 16384 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 Processor -- start 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 -- start start 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f65794cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f657a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a45d0 con 0x561f65784e10 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a6b90 con 0x561f6577f2b0 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a4bd0 con 0x561f6577fca0 2024-03-24T00:44:53.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50126/0 (socket says 172.21.15.123:50126) 2024-03-24T00:44:53.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38c5e00640 1 -- 172.21.15.123:0/3920944324 learned_addr learned my addr 172.21.15.123:0/3920944324 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38c5400640 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f65794cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.225+0000 7f38c6800640 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f657a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 -- 172.21.15.123:0/3920944324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 msgr2=0x561f65794cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f65794cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 -- 172.21.15.123:0/3920944324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 msgr2=0x561f657a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f657a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 -- 172.21.15.123:0/3920944324 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f6578c3f0 con 0x561f6577f2b0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38c5e00640 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f38b401f000 tx=0x7f38b4097a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1838af38020ca8af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38ae600640 1 -- 172.21.15.123:0/3920944324 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40bb040 con 0x561f6577f2b0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38ae600640 1 -- 172.21.15.123:0/3920944324 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38b40860b0 con 0x561f6577f2b0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.226+0000 7f38ae600640 1 -- 172.21.15.123:0/3920944324 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4003c50 con 0x561f6577f2b0 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- 172.21.15.123:0/3920944324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 msgr2=0x561f65779900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f38b401f000 tx=0x7f38b4097a40 comp rx=0 tx=0).stop 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- 172.21.15.123:0/3920944324 shutdown_connections 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f657a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577fca0 0x561f65794cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/3920944324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577f2b0 0x561f65779900 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- 172.21.15.123:0/3920944324 >> 172.21.15.123:0/3920944324 conn(0x561f6577a820 msgr2=0x561f657791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- 172.21.15.123:0/3920944324 shutdown_connections 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- 172.21.15.123:0/3920944324 wait complete. 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.227+0000 7f38cb5aa740 1 -- start start 2024-03-24T00:44:53.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 Processor -- start 2024-03-24T00:44:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577fca0 0x561f65792f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f65793450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a5210 con 0x561f65784e10 2024-03-24T00:44:53.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a6530 con 0x561f6577fca0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01a5870 con 0x561f6577f2b0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38cb5aa740 10 client.0 _ll_register_callbacks cb 0x7f38b40d2450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38c5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577fca0 0x561f65792f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38c5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50038/0 (socket says 172.21.15.123:50038) 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.228+0000 7f38c5e00640 1 -- 172.21.15.123:0/4247702872 learned_addr learned my addr 172.21.15.123:0/4247702872 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c6800640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f65793450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 -- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577fca0 msgr2=0x561f65792f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577fca0 0x561f65792f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 -- 172.21.15.123:0/4247702872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 msgr2=0x561f65793450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f65793450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 -- 172.21.15.123:0/4247702872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f65793990 con 0x561f6577f2b0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38c5e00640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f38b40d2500 tx=0x7f38b400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c01fb380c207fa34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38a3400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40bb040 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38b40860b0 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f38b409acc0 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.229+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38b409f7c0 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 10 client.26380 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561f657f5be0 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38b40860b0 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f38b40b4180 con 0x561f6577f2b0 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 10 client.26380 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 10 client.26380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 20 client.26380 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657f62a0 0x561f657f8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 4 client.26380 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:44:53.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561f656ec740 con 0x561f657f62a0 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.230+0000 7f38c5400640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657f62a0 0x561f657f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.231+0000 7f38c5400640 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657f62a0 0x561f657f8690 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edc7a4c0e50828d4 server_cookie=605565af8bb9de4a in_seq=0 out_seq=0 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.231+0000 7f38a3400640 10 client.26380 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38a3400640 1 -- 172.21.15.123:0/4247702872 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561f656ec740 con 0x561f657f62a0 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38a3400640 10 client.26380 handle_command_reply: tid=0 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 client.26380 shutdown 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 msgr2=0x561f65794900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f38b40d2500 tx=0x7f38b400ef90 comp rx=0 tx=0).stop 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 shutdown_connections 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f65784e10 0x561f65793450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561f6577fca0 0x561f65792f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561f657f62a0 0x561f657f8690 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 --2- 172.21.15.123:0/4247702872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f6577f2b0 0x561f65794900 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38a3400640 0 client.26380 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 >> 172.21.15.123:0/4247702872 conn(0x561f6577a820 msgr2=0x561f65779a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 shutdown_connections 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 1 -- 172.21.15.123:0/4247702872 wait complete. 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.232+0000 7f38cb5aa740 20 client.26380 trim_cache size 0 max 16384 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "2438038b-607b-4554-99cc-f32bcbc40cf1", 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:44:53.237 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 msgr2=0x7f38a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 0x7f38a80a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x561f65780500 tx=0x7f38bc009500 comp rx=0 tx=0).stop 2024-03-24T00:44:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 msgr2=0x7f38c01978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f38b400ea60 tx=0x7f38b400ef20 comp rx=0 tx=0).stop 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 shutdown_connections 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f38a80a2880 0x7f38a80a4d40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0107260 0x7f38c019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0104930 0x7f38c0197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 --2- 172.21.15.123:0/756222155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38c0102000 0x7f38c01978b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 >> 172.21.15.123:0/756222155 conn(0x7f38c00fba10 msgr2=0x7f38c0102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 shutdown_connections 2024-03-24T00:44:53.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.233+0000 7f38c7200640 1 -- 172.21.15.123:0/756222155 wait complete. 2024-03-24T00:44:53.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:44:53.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:44:53.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 -- 172.21.15.123:0/3791646837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 msgr2=0x7f7a9c074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/3791646837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c074500 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009900 tx=0x7f7a8802bcd0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 -- 172.21.15.123:0/3791646837 shutdown_connections 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/3791646837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c109430 0x7f7a9c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/3791646837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c074500 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/3791646837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c075680 0x7f7a9c073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 -- 172.21.15.123:0/3791646837 >> 172.21.15.123:0/3791646837 conn(0x7f7a9c0fcb10 msgr2=0x7f7a9c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 -- 172.21.15.123:0/3791646837 shutdown_connections 2024-03-24T00:44:53.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.448+0000 7f7aa0a00640 1 -- 172.21.15.123:0/3791646837 wait complete. 2024-03-24T00:44:53.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.449+0000 7f7aa0a00640 1 Processor -- start 2024-03-24T00:44:53.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.449+0000 7f7aa0a00640 1 -- start start 2024-03-24T00:44:53.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 0x7f7a9c199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c109430 0x7f7a9c1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c10f470 con 0x7f7a9c109430 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c10f2f0 con 0x7f7a9c0740a0 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7aa0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c10f5f0 con 0x7f7a9c075680 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a99e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 0x7f7a9c199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50142/0 (socket says 172.21.15.123:50142) 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 -- 172.21.15.123:0/1785586444 learned_addr learned my addr 172.21.15.123:0/1785586444 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9b200640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c109430 0x7f7a9c1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 -- 172.21.15.123:0/1785586444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 msgr2=0x7f7a9c199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 0x7f7a9c199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 -- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c109430 msgr2=0x7f7a9c1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c109430 0x7f7a9c1a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a9a800640 1 -- 172.21.15.123:0/1785586444 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c1a1670 con 0x7f7a9c0740a0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.450+0000 7f7a99e00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 0x7f7a9c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7a9a800640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00ea60 tx=0x7f7a8c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7a93400640 1 -- 172.21.15.123:0/1785586444 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c00cde0 con 0x7f7a9c0740a0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7a93400640 1 -- 172.21.15.123:0/1785586444 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c0035f0 con 0x7f7a9c0740a0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7a93400640 1 -- 172.21.15.123:0/1785586444 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c010670 con 0x7f7a9c0740a0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c1a1960 con 0x7f7a9c0740a0 2024-03-24T00:44:53.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.451+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c1a1e30 con 0x7f7a9c0740a0 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.455+0000 7f7aa420b740 1 Processor -- start 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 -- start start 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ad3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a93400640 1 -- 172.21.15.123:0/1785586444 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7a8c0037a0 con 0x7f7a9c0740a0 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432ad6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565432a33250 con 0x565432ad3bb0 2024-03-24T00:44:53.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654329dba30 con 0x565432ace050 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7aa420b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565432a2f2f0 con 0x565432acea40 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50154/0 (socket says 172.21.15.123:50154) 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a9b200640 1 -- 172.21.15.123:0/994975502 learned_addr learned my addr 172.21.15.123:0/994975502 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a9a800640 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ad3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.456+0000 7f7a99e00640 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432ad6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 -- 172.21.15.123:0/994975502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 msgr2=0x565432ad3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ad3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 -- 172.21.15.123:0/994975502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 msgr2=0x565432ad6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432ad6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 -- 172.21.15.123:0/994975502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654329c6690 con 0x565432ace050 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a9b200640 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009fd0 tx=0x7f7a940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e666e1053ee658ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a87400640 1 -- 172.21.15.123:0/994975502 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004d20 con 0x565432ace050 2024-03-24T00:44:53.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a87400640 1 -- 172.21.15.123:0/994975502 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a94004050 con 0x565432ace050 2024-03-24T00:44:53.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.457+0000 7f7a87400640 1 -- 172.21.15.123:0/994975502 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94005260 con 0x565432ace050 2024-03-24T00:44:53.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7a93400640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 0x7f7a800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 -- 172.21.15.123:0/994975502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 msgr2=0x565432ace470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009fd0 tx=0x7f7a940118d0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7a93400640 1 -- 172.21.15.123:0/1785586444 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7a8c0ced30 con 0x7f7a9c0740a0 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7a9a800640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 0x7f7a800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 -- 172.21.15.123:0/994975502 shutdown_connections 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432ad6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ad3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 --2- 172.21.15.123:0/994975502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432ace470 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 -- 172.21.15.123:0/994975502 >> 172.21.15.123:0/994975502 conn(0x565432a96200 msgr2=0x565432ac99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.458+0000 7f7aa420b740 1 -- 172.21.15.123:0/994975502 shutdown_connections 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 -- 172.21.15.123:0/994975502 wait complete. 2024-03-24T00:44:53.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7a9a800640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 0x7f7a800a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x565432acee40 tx=0x7f7a8c0d2800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 -- start start 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 Processor -- start 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.459+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7aa420b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x565432ad6b60 con 0x565432acea40 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7aa420b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654329eee80 con 0x565432ad3bb0 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7aa420b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x565432ad6ce0 con 0x565432ace050 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7aa420b740 10 client.0 _ll_register_callbacks cb 0x565432a01130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50064/0 (socket says 172.21.15.123:50064) 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 -- 172.21.15.123:0/2221754224 learned_addr learned my addr 172.21.15.123:0/2221754224 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 -- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 msgr2=0x565432ae7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 -- 172.21.15.123:0/2221754224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 msgr2=0x565432ae0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.460+0000 7f7a9b200640 1 -- 172.21.15.123:0/2221754224 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654328117d0 con 0x565432ace050 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a9b200640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7a9401b040 tx=0x7f7a94009c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b82ecdd534ba9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a7be00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400f4e0 con 0x565432ace050 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7a9400f680 con 0x565432ace050 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a94004050 con 0x565432ace050 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a9400f880 con 0x565432ace050 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7aa420b740 10 client.26392 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:53.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5654327473f0 con 0x565432ace050 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.461+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a94004050 con 0x565432ace050 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7a94025040 con 0x565432ace050 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 10 client.26392 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 10 client.26392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 20 client.26392 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432aec460 0x565432aee850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 4 client.26392 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565432a40790 con 0x565432aec460 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.462+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432aec460 0x565432aee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7a7be00640 10 client.26392 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432aec460 0x565432aee850 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b42478d13c60e708 server_cookie=d378b9c4e7f8eb5a in_seq=0 out_seq=0 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7a7be00640 1 -- 172.21.15.123:0/2221754224 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565432a40790 con 0x565432aec460 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7a7be00640 10 client.26392 handle_command_reply: tid=0 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7aa420b740 1 client.26392 shutdown 2024-03-24T00:44:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 msgr2=0x565432ae0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.463+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7a9401b040 tx=0x7f7a94009c80 comp rx=0 tx=0).stop 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 shutdown_connections 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432aec460 0x565432aee850 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2221754224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae0840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 >> 172.21.15.123:0/2221754224 conn(0x565432a96200 msgr2=0x565432ac8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7a7be00640 0 client.26392 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 shutdown_connections 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 1 -- 172.21.15.123:0/2221754224 wait complete. 2024-03-24T00:44:53.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.464+0000 7f7aa420b740 20 client.26392 trim_cache size 0 max 16384 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 Processor -- start 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 -- start start 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432acd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432af30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a46f0 con 0x565432ad3bb0 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a6c90 con 0x565432ace050 2024-03-24T00:44:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7aa420b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a4cf0 con 0x565432acea40 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432acd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7a99e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432acd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50168/0 (socket says 172.21.15.123:50168) 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7a9b200640 1 -- 172.21.15.123:0/2044484922 learned_addr learned my addr 172.21.15.123:0/2044484922 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.481+0000 7f7a99e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50066/0 (socket says 172.21.15.123:50066) 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a9b200640 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432af30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 -- 172.21.15.123:0/2044484922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 msgr2=0x565432acd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432acd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 -- 172.21.15.123:0/2044484922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 msgr2=0x565432af30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432af30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 -- 172.21.15.123:0/2044484922 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654327070f0 con 0x565432acea40 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a99e00640 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7a88036040 tx=0x7f7a88009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d6d5d737877b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a86a00640 1 -- 172.21.15.123:0/2044484922 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8804d070 con 0x565432acea40 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.482+0000 7f7a86a00640 1 -- 172.21.15.123:0/2044484922 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a88004b90 con 0x565432acea40 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7a86a00640 1 -- 172.21.15.123:0/2044484922 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88004f30 con 0x565432acea40 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 -- 172.21.15.123:0/2044484922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 msgr2=0x565432ae7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7a88036040 tx=0x7f7a88009260 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 -- 172.21.15.123:0/2044484922 shutdown_connections 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432ad3bb0 0x565432af30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432acea40 0x565432ae7d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 --2- 172.21.15.123:0/2044484922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ace050 0x565432acd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 -- 172.21.15.123:0/2044484922 >> 172.21.15.123:0/2044484922 conn(0x565432a96200 msgr2=0x565432accc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 -- 172.21.15.123:0/2044484922 shutdown_connections 2024-03-24T00:44:53.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.483+0000 7f7aa420b740 1 -- 172.21.15.123:0/2044484922 wait complete. 2024-03-24T00:44:53.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 -- start start 2024-03-24T00:44:53.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 Processor -- start 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a5310 con 0x565432acea40 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a6630 con 0x565432ad3bb0 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1a5970 con 0x565432ace050 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.484+0000 7f7aa420b740 10 client.0 _ll_register_callbacks cb 0x7f7a8c0d26d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50076/0 (socket says 172.21.15.123:50076) 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 -- 172.21.15.123:0/1502136333 learned_addr learned my addr 172.21.15.123:0/1502136333 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a99e00640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 -- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 msgr2=0x565432ae72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 -- 172.21.15.123:0/1502136333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 msgr2=0x565432ae62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 -- 172.21.15.123:0/1502136333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565432ae78b0 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a99e00640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a9b200640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7a94015010 tx=0x7f7a94004b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed0eeb6da8ccdb73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a7be00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94003970 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a88004b90 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.485+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7a94004810 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a9400fa20 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 10 client.26398 fetch_fsmap learned FSMap version 19 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565432b48570 con 0x565432ace050 2024-03-24T00:44:53.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a88004b90 con 0x565432ace050 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7a94019620 con 0x565432ace050 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 10 client.26398 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 10 client.26398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 20 client.26398 populate_metadata read hostname 'smithi123' 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432b48db0 0x565432b4b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 4 client.26398 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565432a3aae0 con 0x565432b48db0 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.486+0000 7f7a99e00640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432b48db0 0x565432b4b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7a99e00640 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432b48db0 0x565432b4b1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc2134e29fc896f server_cookie=987204a7e959b87a in_seq=0 out_seq=0 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7a7be00640 10 client.26398 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7a7be00640 1 -- 172.21.15.123:0/1502136333 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+267 (crc 0 0 0) 0x565432a3aae0 con 0x565432b48db0 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7a7be00640 10 client.26398 handle_command_reply: tid=0 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 client.26398 shutdown 2024-03-24T00:44:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 msgr2=0x565432ae5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7a94015010 tx=0x7f7a94004b60 comp rx=0 tx=0).stop 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 shutdown_connections 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565432ad3bb0 0x565432ae72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x565432acea40 0x565432ae62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x565432b48db0 0x565432b4b1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 --2- 172.21.15.123:0/1502136333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565432ace050 0x565432ae5d70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7a7be00640 0 client.26398 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.487+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 >> 172.21.15.123:0/1502136333 conn(0x565432a96200 msgr2=0x565432ac8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 shutdown_connections 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa420b740 1 -- 172.21.15.123:0/1502136333 wait complete. 2024-03-24T00:44:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa420b740 20 client.26398 trim_cache size 0 max 16384 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (222 inodes in the stack)", 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "2438038b-607b-4554-99cc-f32bcbc40cf1": { 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "2438038b-607b-4554-99cc-f32bcbc40cf1", 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:44:53.494 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 msgr2=0x7f7a800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 0x7f7a800a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x565432acee40 tx=0x7f7a8c0d2800 comp rx=0 tx=0).stop 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 msgr2=0x7f7a9c199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00ea60 tx=0x7f7a8c00ef20 comp rx=0 tx=0).stop 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 shutdown_connections 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7a800a2800 0x7f7a800a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a9c109430 0x7f7a9c1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a9c075680 0x7f7a9c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 --2- 172.21.15.123:0/1785586444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a9c0740a0 0x7f7a9c199a50 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.488+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 >> 172.21.15.123:0/1785586444 conn(0x7f7a9c0fcb10 msgr2=0x7f7a9c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.489+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 shutdown_connections 2024-03-24T00:44:53.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:44:53.489+0000 7f7aa0a00640 1 -- 172.21.15.123:0/1785586444 wait complete. 2024-03-24T00:44:53.505 INFO:tasks.cephfs.filesystem:scrub status for tag:2438038b-607b-4554-99cc-f32bcbc40cf1 - {'path': '/', 'tag': '2438038b-607b-4554-99cc-f32bcbc40cf1', 'options': 'recursive,force'} 2024-03-24T00:44:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:54 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:44:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:54 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:54.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:54 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:44:54.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:54 smithi177 ceph-mon[36034]: pgmap v595: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 40 MiB/s wr, 3.67k op/s 2024-03-24T00:44:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:54 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:44:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:54 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:54 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:44:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:54 smithi204 ceph-mon[36345]: pgmap v595: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 40 MiB/s wr, 3.67k op/s 2024-03-24T00:44:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:54 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:44:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:54 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:54 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:44:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:54 smithi123 ceph-mon[31487]: pgmap v595: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 40 MiB/s wr, 3.67k op/s 2024-03-24T00:44:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:56 smithi177 ceph-mon[36034]: pgmap v596: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:56 smithi204 ceph-mon[36345]: pgmap v596: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:56 smithi123 ceph-mon[31487]: pgmap v596: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 40 MiB/s wr, 3.78k op/s 2024-03-24T00:44:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:58 smithi177 ceph-mon[36034]: pgmap v597: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 37 MiB/s wr, 3.50k op/s 2024-03-24T00:44:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:58 smithi204 ceph-mon[36345]: pgmap v597: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 37 MiB/s wr, 3.50k op/s 2024-03-24T00:44:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:58 smithi123 ceph-mon[31487]: pgmap v597: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 37 MiB/s wr, 3.50k op/s 2024-03-24T00:44:59.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:44:59 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:59.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:44:59 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:44:59.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:44:59 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:00 smithi177 ceph-mon[36034]: pgmap v598: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:45:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:00 smithi204 ceph-mon[36345]: pgmap v598: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:45:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:00 smithi123 ceph-mon[31487]: pgmap v598: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 35 MiB/s wr, 3.48k op/s 2024-03-24T00:45:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:02 smithi177 ceph-mon[36034]: pgmap v599: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 34 MiB/s wr, 3.31k op/s 2024-03-24T00:45:02.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:02 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:45:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:02 smithi204 ceph-mon[36345]: pgmap v599: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 34 MiB/s wr, 3.31k op/s 2024-03-24T00:45:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:02 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:45:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:02 smithi123 ceph-mon[31487]: pgmap v599: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 34 MiB/s wr, 3.31k op/s 2024-03-24T00:45:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:02 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:45:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 -- 172.21.15.123:0/4081287263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c101510 msgr2=0x7f6b2c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 --2- 172.21.15.123:0/4081287263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c101510 0x7f6b2c101930 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009900 tx=0x7f6b1c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 -- 172.21.15.123:0/4081287263 shutdown_connections 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 --2- 172.21.15.123:0/4081287263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 --2- 172.21.15.123:0/4081287263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c102700 0x7f6b2c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 --2- 172.21.15.123:0/4081287263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c101510 0x7f6b2c101930 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 -- 172.21.15.123:0/4081287263 >> 172.21.15.123:0/4081287263 conn(0x7f6b2c0fca80 msgr2=0x7f6b2c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 -- 172.21.15.123:0/4081287263 shutdown_connections 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.349+0000 7f6b2b200640 1 -- 172.21.15.123:0/4081287263 wait complete. 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 Processor -- start 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 -- start start 2024-03-24T00:45:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 0x7f6b2c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c10d1f0 con 0x7f6b2c1030c0 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c10d070 con 0x7f6b2c102700 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b2b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c10d370 con 0x7f6b2c101510 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:36352/0 (socket says 172.21.15.123:36352) 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 0x7f6b2c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.350+0000 7f6b29400640 1 -- 172.21.15.123:0/2186013302 learned_addr learned my addr 172.21.15.123:0/2186013302 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b2a800640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 -- 172.21.15.123:0/2186013302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 msgr2=0x7f6b2c197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 0x7f6b2c197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 -- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 msgr2=0x7f6b2c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 -- 172.21.15.123:0/2186013302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c19f460 con 0x7f6b2c102700 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b2a800640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 0x7f6b2c197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b29400640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e9f0 tx=0x7f6b1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b22a00640 1 -- 172.21.15.123:0/2186013302 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800cd80 con 0x7f6b2c102700 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b22a00640 1 -- 172.21.15.123:0/2186013302 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b18005500 con 0x7f6b2c102700 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b22a00640 1 -- 172.21.15.123:0/2186013302 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18010690 con 0x7f6b2c102700 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.351+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c19f750 con 0x7f6b2c102700 2024-03-24T00:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.352+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c19fc50 con 0x7f6b2c102700 2024-03-24T00:45:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.355+0000 7f6b22a00640 1 -- 172.21.15.123:0/2186013302 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6b180046c0 con 0x7f6b2c102700 2024-03-24T00:45:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.355+0000 7f6b33c51740 1 Processor -- start 2024-03-24T00:45:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.355+0000 7f6b33c51740 1 -- start start 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f7f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f7f93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f7b43b0 con 0x55633f7f6f50 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f756830 con 0x55633f7f1de0 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b33c51740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f6ff5d0 con 0x55633f7f13f0 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46136/0 (socket says 172.21.15.123:46136) 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 -- 172.21.15.123:0/452780692 learned_addr learned my addr 172.21.15.123:0/452780692 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b29e00640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f7f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b29400640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f7f93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 -- 172.21.15.123:0/452780692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 msgr2=0x55633f7f68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f7f68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 -- 172.21.15.123:0/452780692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 msgr2=0x55633f7f93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f7f93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b2a800640 1 -- 172.21.15.123:0/452780692 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633f7172c0 con 0x55633f7f13f0 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.356+0000 7f6b29400640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f7f93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b2a800640 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009fd0 tx=0x7f6b240108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7caaf24203639533 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b22a00640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 0x7f6b100a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b17400640 1 -- 172.21.15.123:0/452780692 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24011760 con 0x55633f7f13f0 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b17400640 1 -- 172.21.15.123:0/452780692 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b24011de0 con 0x55633f7f13f0 2024-03-24T00:45:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b17400640 1 -- 172.21.15.123:0/452780692 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400e3c0 con 0x55633f7f13f0 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b22a00640 1 -- 172.21.15.123:0/2186013302 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b180cef50 con 0x7f6b2c102700 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.357+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 0x7f6b100a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 -- 172.21.15.123:0/452780692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 msgr2=0x55633f7f1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009fd0 tx=0x7f6b240108e0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 -- 172.21.15.123:0/452780692 shutdown_connections 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f7f93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f7f68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 --2- 172.21.15.123:0/452780692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f7f1810 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 0x7f6b100a4cb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02c180 tx=0x7f6b1c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 -- 172.21.15.123:0/452780692 >> 172.21.15.123:0/452780692 conn(0x55633f7ec960 msgr2=0x55633f7ecd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 -- 172.21.15.123:0/452780692 shutdown_connections 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.358+0000 7f6b33c51740 1 -- 172.21.15.123:0/452780692 wait complete. 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.359+0000 7f6b33c51740 1 -- start start 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.359+0000 7f6b33c51740 1 Processor -- start 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.359+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.359+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f802b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b33c51740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f712a20 con 0x55633f7f6f50 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b33c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f7528d0 con 0x55633f7f13f0 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b33c51740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f7f9f00 con 0x55633f7f1de0 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b33c51740 10 client.0 _ll_register_callbacks cb 0x55633f74f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:36370/0 (socket says 172.21.15.123:36370) 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 -- 172.21.15.123:0/2931048458 learned_addr learned my addr 172.21.15.123:0/2931048458 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29400640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b2a800640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f802b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 -- 172.21.15.123:0/2931048458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 msgr2=0x55633f7fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7fbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 -- 172.21.15.123:0/2931048458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 msgr2=0x55633f802b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f802b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29e00640 1 -- 172.21.15.123:0/2931048458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55633f46b210 con 0x55633f7f13f0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.360+0000 7f6b29400640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b29e00640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02c8f0 tx=0x7f6b1c02bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb27d1fe1a259ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b0be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c005380 con 0x55633f7f13f0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b1c005520 con 0x55633f7f13f0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b24011de0 con 0x55633f7f13f0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b1c0035f0 con 0x55633f7f13f0 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b33c51740 10 client.26396 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55633f537620 con 0x55633f7f13f0 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.361+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b24011de0 con 0x55633f7f13f0 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6b1c05c020 con 0x55633f7f13f0 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 10 client.26396 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 10 client.26396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 20 client.26396 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f807350 0x55633f809760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 4 client.26396 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55633f763d70 con 0x55633f807350 2024-03-24T00:45:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b29400640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f807350 0x55633f809760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b29400640 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f807350 0x55633f809760 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24887594639d618d server_cookie=2d91f3bddbeb60f7 in_seq=0 out_seq=0 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.362+0000 7f6b0be00640 10 client.26396 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.363+0000 7f6b0be00640 1 -- 172.21.15.123:0/2931048458 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55633f763d70 con 0x55633f807350 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.363+0000 7f6b0be00640 10 client.26396 handle_command_reply: tid=0 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 client.26396 shutdown 2024-03-24T00:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 msgr2=0x55633f7fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02c8f0 tx=0x7f6b1c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 shutdown_connections 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f802b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f807350 0x55633f809760 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 --2- 172.21.15.123:0/2931048458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f13f0 0x55633f7fb5e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 >> 172.21.15.123:0/2931048458 conn(0x55633f7ec960 msgr2=0x55633f7ebb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b0be00640 0 client.26396 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 shutdown_connections 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 1 -- 172.21.15.123:0/2931048458 wait complete. 2024-03-24T00:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.364+0000 7f6b33c51740 20 client.26396 trim_cache size 0 max 16384 2024-03-24T00:45:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.381+0000 7f6b33c51740 1 Processor -- start 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.381+0000 7f6b33c51740 1 -- start start 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7e76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f6f50 0x55633f7ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a69b0 con 0x55633f7f13f0 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a8f70 con 0x55633f7f6f50 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b33c51740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a6fb0 con 0x55633f7f1de0 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b2a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f6f50 0x55633f7ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33574/0 (socket says 172.21.15.123:33574) 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7e76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29e00640 1 -- 172.21.15.123:0/4123795881 learned_addr learned my addr 172.21.15.123:0/4123795881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 -- 172.21.15.123:0/4123795881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f6f50 msgr2=0x55633f7ecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f6f50 0x55633f7ecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 -- 172.21.15.123:0/4123795881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 msgr2=0x55633f7fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29400640 1 -- 172.21.15.123:0/4123795881 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633f7f17d0 con 0x55633f7f1de0 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.382+0000 7f6b29e00640 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.383+0000 7f6b29400640 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7e76a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6b1809b830 tx=0x7f6b180cc8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=143a8cf6cf0f7acd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.383+0000 7f6b16a00640 1 -- 172.21.15.123:0/4123795881 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1809b8a0 con 0x55633f7f1de0 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.383+0000 7f6b16a00640 1 -- 172.21.15.123:0/4123795881 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1809f9f0 con 0x55633f7f1de0 2024-03-24T00:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.383+0000 7f6b16a00640 1 -- 172.21.15.123:0/4123795881 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1809bd60 con 0x55633f7f1de0 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- 172.21.15.123:0/4123795881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 msgr2=0x55633f7e76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7e76a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6b1809b830 tx=0x7f6b180cc8d0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- 172.21.15.123:0/4123795881 shutdown_connections 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f6f50 0x55633f7ecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f1de0 0x55633f7e76a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 --2- 172.21.15.123:0/4123795881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f13f0 0x55633f7fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- 172.21.15.123:0/4123795881 >> 172.21.15.123:0/4123795881 conn(0x55633f7ec960 msgr2=0x55633f7f2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- 172.21.15.123:0/4123795881 shutdown_connections 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- 172.21.15.123:0/4123795881 wait complete. 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.384+0000 7f6b33c51740 1 -- start start 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 Processor -- start 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f85df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f8654f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a75f0 con 0x55633f7f6f50 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a8910 con 0x55633f7f1de0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1a7c50 con 0x55633f7f13f0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b33c51740 10 client.0 _ll_register_callbacks cb 0x7f6b1c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:36384/0 (socket says 172.21.15.123:36384) 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b29e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f85df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b29400640 1 -- 172.21.15.123:0/8564824 learned_addr learned my addr 172.21.15.123:0/8564824 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.385+0000 7f6b2a800640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f8654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 -- 172.21.15.123:0/8564824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 msgr2=0x55633f85df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f85df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 -- 172.21.15.123:0/8564824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 msgr2=0x55633f8654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f8654f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 -- 172.21.15.123:0/8564824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55633f865ae0 con 0x55633f7f1de0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b29400640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6b180097d0 tx=0x7f6b1809b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e03c87f7cb39092 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b0be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b180af060 con 0x55633f7f1de0 2024-03-24T00:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b1809cbd0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b1809f9f0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b1809cdd0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b33c51740 10 client.26402 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.386+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55633f866df0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1809f9f0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6b1809acf0 con 0x55633f7f1de0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 10 client.26402 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 10 client.26402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 20 client.26402 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f867720 0x55633f869b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 4 client.26402 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55633f75e0c0 con 0x55633f867720 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.387+0000 7f6b29e00640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f867720 0x55633f869b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.388+0000 7f6b29e00640 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f867720 0x55633f869b40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90ac906a1fcba4e server_cookie=f073cbe2c2d1e166 in_seq=0 out_seq=0 2024-03-24T00:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:04.388+0000 7f6b0be00640 10 client.26402 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:04 smithi177 ceph-mon[36034]: pgmap v600: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 32 MiB/s wr, 3.08k op/s 2024-03-24T00:45:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:04 smithi204 ceph-mon[36345]: pgmap v600: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 32 MiB/s wr, 3.08k op/s 2024-03-24T00:45:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:04 smithi123 ceph-mon[31487]: pgmap v600: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 32 MiB/s wr, 3.08k op/s 2024-03-24T00:45:06.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:06 smithi177 ceph-mon[36034]: pgmap v601: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 34 MiB/s wr, 3.11k op/s 2024-03-24T00:45:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:06 smithi204 ceph-mon[36345]: pgmap v601: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 34 MiB/s wr, 3.11k op/s 2024-03-24T00:45:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:06 smithi123 ceph-mon[31487]: pgmap v601: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 34 MiB/s wr, 3.11k op/s 2024-03-24T00:45:08.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:08 smithi177 ceph-mon[36034]: pgmap v602: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 32 MiB/s wr, 2.85k op/s 2024-03-24T00:45:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:08 smithi204 ceph-mon[36345]: pgmap v602: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 32 MiB/s wr, 2.85k op/s 2024-03-24T00:45:08.858 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:08 smithi123 ceph-mon[31487]: pgmap v602: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 32 MiB/s wr, 2.85k op/s 2024-03-24T00:45:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:10 smithi177 ceph-mon[36034]: pgmap v603: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 32 MiB/s wr, 2.94k op/s 2024-03-24T00:45:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:10 smithi204 ceph-mon[36345]: pgmap v603: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 32 MiB/s wr, 2.94k op/s 2024-03-24T00:45:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:10 smithi123 ceph-mon[31487]: pgmap v603: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 32 MiB/s wr, 2.94k op/s 2024-03-24T00:45:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:12 smithi177 ceph-mon[36034]: pgmap v604: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 36 MiB/s wr, 3.19k op/s 2024-03-24T00:45:12.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:12 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:12 smithi204 ceph-mon[36345]: pgmap v604: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 36 MiB/s wr, 3.19k op/s 2024-03-24T00:45:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:12 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:12.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:12 smithi123 ceph-mon[31487]: pgmap v604: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 36 MiB/s wr, 3.19k op/s 2024-03-24T00:45:12.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:12 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:13.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:13 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:45:13.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:13.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:13.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:13 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:45:13.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:45:13.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:13.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:13.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:13 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:45:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:14 smithi177 ceph-mon[36034]: Adjusting osd_memory_target on smithi204 to 174.1M 2024-03-24T00:45:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:14 smithi177 ceph-mon[36034]: Unable to set osd_memory_target on smithi204 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-24T00:45:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:14 smithi177 ceph-mon[36034]: pgmap v605: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 3.32k op/s 2024-03-24T00:45:14.758 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:14 smithi123 ceph-mon[31487]: Adjusting osd_memory_target on smithi204 to 174.1M 2024-03-24T00:45:14.758 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:14 smithi123 ceph-mon[31487]: Unable to set osd_memory_target on smithi204 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-24T00:45:14.758 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:14 smithi123 ceph-mon[31487]: pgmap v605: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 3.32k op/s 2024-03-24T00:45:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:14 smithi204 ceph-mon[36345]: Adjusting osd_memory_target on smithi204 to 174.1M 2024-03-24T00:45:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:14 smithi204 ceph-mon[36345]: Unable to set osd_memory_target on smithi204 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-24T00:45:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:14 smithi204 ceph-mon[36345]: pgmap v605: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 38 MiB/s wr, 3.32k op/s 2024-03-24T00:45:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:16 smithi204 ceph-mon[36345]: pgmap v606: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 37 MiB/s wr, 3.18k op/s 2024-03-24T00:45:16.859 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:16 smithi123 ceph-mon[31487]: pgmap v606: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 37 MiB/s wr, 3.18k op/s 2024-03-24T00:45:16.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:16 smithi177 ceph-mon[36034]: pgmap v606: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 37 MiB/s wr, 3.18k op/s 2024-03-24T00:45:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:17 smithi204 ceph-mon[36345]: pgmap v607: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 35 MiB/s wr, 3.10k op/s 2024-03-24T00:45:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:17 smithi123 ceph-mon[31487]: pgmap v607: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 35 MiB/s wr, 3.10k op/s 2024-03-24T00:45:17.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:17 smithi177 ceph-mon[36034]: pgmap v607: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 35 MiB/s wr, 3.10k op/s 2024-03-24T00:45:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.696+0000 7f6b0be00640 1 -- 172.21.15.123:0/8564824 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55633f75e0c0 con 0x55633f867720 2024-03-24T00:45:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.696+0000 7f6b0be00640 10 client.26402 handle_command_reply: tid=0 2024-03-24T00:45:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.696+0000 7f6b33c51740 1 client.26402 shutdown 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.696+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 msgr2=0x55633f85e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.696+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6b180097d0 tx=0x7f6b1809b8a0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 shutdown_connections 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55633f7f6f50 0x55633f8654f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55633f7f1de0 0x55633f85e4b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55633f867720 0x55633f869b40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 --2- 172.21.15.123:0/8564824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55633f7f13f0 0x55633f85df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 >> 172.21.15.123:0/8564824 conn(0x55633f7ec960 msgr2=0x55633f7fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b0be00640 0 client.26402 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 shutdown_connections 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 1 -- 172.21.15.123:0/8564824 wait complete. 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.697+0000 7f6b33c51740 20 client.26402 trim_cache size 0 max 16384 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:45:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:45:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:45:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:45:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.698+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 msgr2=0x7f6b100a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.698+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 0x7f6b100a4cb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02c180 tx=0x7f6b1c02cf90 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.698+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 msgr2=0x7f6b2c197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.698+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e9f0 tx=0x7f6b1800edf0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 shutdown_connections 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6b100a27f0 0x7f6b100a4cb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c1030c0 0x7f6b2c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b2c102700 0x7f6b2c197d80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 --2- 172.21.15.123:0/2186013302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b2c101510 0x7f6b2c197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.699+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 >> 172.21.15.123:0/2186013302 conn(0x7f6b2c0fca80 msgr2=0x7f6b2c105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.700+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 shutdown_connections 2024-03-24T00:45:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:20.700+0000 7f6b2b200640 1 -- 172.21.15.123:0/2186013302 wait complete. 2024-03-24T00:45:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:20 smithi177 ceph-mon[36034]: pgmap v608: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 34 MiB/s wr, 2.97k op/s 2024-03-24T00:45:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:20 smithi204 ceph-mon[36345]: pgmap v608: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 34 MiB/s wr, 2.97k op/s 2024-03-24T00:45:20.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:20 smithi123 ceph-mon[31487]: pgmap v608: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 34 MiB/s wr, 2.97k op/s 2024-03-24T00:45:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:22 smithi177 ceph-mon[36034]: pgmap v609: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 39 MiB/s wr, 3.43k op/s 2024-03-24T00:45:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:22 smithi204 ceph-mon[36345]: pgmap v609: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 39 MiB/s wr, 3.43k op/s 2024-03-24T00:45:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:22 smithi123 ceph-mon[31487]: pgmap v609: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 39 MiB/s wr, 3.43k op/s 2024-03-24T00:45:23.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:45:23.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:45:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.714+0000 7f5c5fe00640 1 -- 172.21.15.123:0/2583535747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60100480 msgr2=0x7f5c601008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.714+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/2583535747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60100480 0x7f5c601008a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009900 tx=0x7f5c5002bc40 comp rx=0 tx=0).stop 2024-03-24T00:45:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 -- 172.21.15.123:0/2583535747 shutdown_connections 2024-03-24T00:45:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/2583535747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 0x7f5c601088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/2583535747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60101670 0x7f5c60101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/2583535747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60100480 0x7f5c601008a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 -- 172.21.15.123:0/2583535747 >> 172.21.15.123:0/2583535747 conn(0x7f5c600fba10 msgr2=0x7f5c600fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 -- 172.21.15.123:0/2583535747 shutdown_connections 2024-03-24T00:45:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.715+0000 7f5c5fe00640 1 -- 172.21.15.123:0/2583535747 wait complete. 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 Processor -- start 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 -- start start 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 0x7f5c60102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 0x7f5c60102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6010c130 con 0x7f5c60102030 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6010bfb0 con 0x7f5c60101670 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6010c2b0 con 0x7f5c60100480 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50158/0 (socket says 172.21.15.123:50158) 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.716+0000 7f5c5ea00640 1 -- 172.21.15.123:0/960047559 learned_addr learned my addr 172.21.15.123:0/960047559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5e000640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 0x7f5c60102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5f400640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 0x7f5c60102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 -- 172.21.15.123:0/960047559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 msgr2=0x7f5c60102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 0x7f5c60102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 -- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 msgr2=0x7f5c60102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 0x7f5c60102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 -- 172.21.15.123:0/960047559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60103820 con 0x7f5c60100480 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5e000640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 0x7f5c60102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.717+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5c5002efd0 tx=0x7f5c5002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.718+0000 7f5c57400640 1 -- 172.21.15.123:0/960047559 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50041070 con 0x7f5c60100480 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.718+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c601a1940 con 0x7f5c60100480 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.718+0000 7f5c57400640 1 -- 172.21.15.123:0/960047559 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c50004450 con 0x7f5c60100480 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.718+0000 7f5c57400640 1 -- 172.21.15.123:0/960047559 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50003e50 con 0x7f5c60100480 2024-03-24T00:45:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.718+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c601a1eb0 con 0x7f5c60100480 2024-03-24T00:45:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.721+0000 7f5c57400640 1 -- 172.21.15.123:0/960047559 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5c50044070 con 0x7f5c60100480 2024-03-24T00:45:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.721+0000 7f5c687d3740 1 Processor -- start 2024-03-24T00:45:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.721+0000 7f5c687d3740 1 -- start start 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343d9e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d9ebb0 0x557343da1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343cfe250 con 0x557343d99a40 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343ca6a30 con 0x557343d99050 2024-03-24T00:45:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c687d3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343cfa2f0 con 0x557343d9ebb0 2024-03-24T00:45:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343d9e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39038/0 (socket says 172.21.15.123:39038) 2024-03-24T00:45:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 -- 172.21.15.123:0/80169744 learned_addr learned my addr 172.21.15.123:0/80169744 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c57400640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 0x7f5c440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 -- 172.21.15.123:0/80169744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d9ebb0 msgr2=0x557343da1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d9ebb0 0x557343da1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 -- 172.21.15.123:0/80169744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 msgr2=0x557343d9e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.722+0000 7f5c5f400640 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343d9e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c5f400640 1 -- 172.21.15.123:0/80169744 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557343c91690 con 0x557343d99050 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c57400640 1 -- 172.21.15.123:0/960047559 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5c500f0700 con 0x7f5c60100480 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c5f400640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 0x7f5c440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c5f400640 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009fc0 tx=0x7f5c58010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb0ebaa27bc6ddb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c4fe00640 1 -- 172.21.15.123:0/80169744 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58004040 con 0x557343d99050 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c4fe00640 1 -- 172.21.15.123:0/80169744 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c58004740 con 0x557343d99050 2024-03-24T00:45:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.723+0000 7f5c4fe00640 1 -- 172.21.15.123:0/80169744 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5801e680 con 0x557343d99050 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 -- 172.21.15.123:0/80169744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 msgr2=0x557343d99470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c5f400640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 0x7f5c440a4e60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5c5801f000 tx=0x7f5c5801ed80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009fc0 tx=0x7f5c58010860 comp rx=0 tx=0).stop 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 -- 172.21.15.123:0/80169744 shutdown_connections 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d9ebb0 0x557343da1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343d9e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 --2- 172.21.15.123:0/80169744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343d99470 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 -- 172.21.15.123:0/80169744 >> 172.21.15.123:0/80169744 conn(0x557343d61200 msgr2=0x557343d949c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.724+0000 7f5c687d3740 1 -- 172.21.15.123:0/80169744 shutdown_connections 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.725+0000 7f5c687d3740 1 -- 172.21.15.123:0/80169744 wait complete. 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.725+0000 7f5c687d3740 1 -- start start 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.725+0000 7f5c687d3740 1 Processor -- start 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343da3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343da1b60 con 0x557343d99a40 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343cb9e80 con 0x557343d9ebb0 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557343da1ce0 con 0x557343d99050 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c687d3740 10 client.0 _ll_register_callbacks cb 0x557343ccc130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39050/0 (socket says 172.21.15.123:39050) 2024-03-24T00:45:23.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 -- 172.21.15.123:0/955555991 learned_addr learned my addr 172.21.15.123:0/955555991 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343da3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 -- 172.21.15.123:0/955555991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 msgr2=0x557343da3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343da3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 -- 172.21.15.123:0/955555991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 msgr2=0x557343da3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.726+0000 7f5c5f400640 1 -- 172.21.15.123:0/955555991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557343adc7d0 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c5f400640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5c58003f60 tx=0x7f5c58017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85079ac95869ed19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58037060 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c58004740 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5c5801e420 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c5801e620 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c687d3740 10 client.26414 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.727+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557343a123f0 con 0x557343d9ebb0 2024-03-24T00:45:23.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c58004740 con 0x557343d9ebb0 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5c58004040 con 0x557343d9ebb0 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 10 client.26414 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 10 client.26414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 20 client.26414 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343daebc0 0x557343db0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 4 client.26414 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557343d0b790 con 0x557343daebc0 2024-03-24T00:45:23.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.728+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343daebc0 0x557343db0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.729+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343daebc0 0x557343db0fb0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b6ef696764411dc server_cookie=eb2c28ec13b85b95 in_seq=0 out_seq=0 2024-03-24T00:45:23.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.729+0000 7f5c4cc00640 10 client.26414 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:23.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.729+0000 7f5c4cc00640 1 -- 172.21.15.123:0/955555991 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557343d0b790 con 0x557343daebc0 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.729+0000 7f5c4cc00640 10 client.26414 handle_command_reply: tid=0 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.729+0000 7f5c687d3740 1 client.26414 shutdown 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 msgr2=0x557343daa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5c58003f60 tx=0x7f5c58017000 comp rx=0 tx=0).stop 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 shutdown_connections 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d9ebb0 0x557343daa5b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343daebc0 0x557343db0fb0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 --2- 172.21.15.123:0/955555991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343da3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c4cc00640 0 client.26414 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 >> 172.21.15.123:0/955555991 conn(0x557343d61200 msgr2=0x557343d93bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 shutdown_connections 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 1 -- 172.21.15.123:0/955555991 wait complete. 2024-03-24T00:45:23.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.730+0000 7f5c687d3740 20 client.26414 trim_cache size 0 max 16384 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 Processor -- start 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 -- start start 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d61200 0x557343d93a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 0x557343daa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601a8b30 con 0x557343d61200 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601ab0f0 con 0x557343d99a40 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c687d3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601a9130 con 0x557343d99050 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c5e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c5f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 0x557343daa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c5e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50188/0 (socket says 172.21.15.123:50188) 2024-03-24T00:45:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c5e000640 1 -- 172.21.15.123:0/2026663269 learned_addr learned my addr 172.21.15.123:0/2026663269 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.749+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d61200 0x557343d93a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 -- 172.21.15.123:0/2026663269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 msgr2=0x557343d93f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 -- 172.21.15.123:0/2026663269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d61200 msgr2=0x557343d93a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d61200 0x557343d93a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 -- 172.21.15.123:0/2026663269 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573439d20f0 con 0x557343d99a40 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5e000640 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c5f400640 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 0x557343daa570 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5c58011010 tx=0x7f5c580097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483a6d6446df1b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c4f400640 1 -- 172.21.15.123:0/2026663269 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58042070 con 0x557343d99a40 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c4f400640 1 -- 172.21.15.123:0/2026663269 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c58009e90 con 0x557343d99a40 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.750+0000 7f5c4f400640 1 -- 172.21.15.123:0/2026663269 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58037070 con 0x557343d99a40 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 -- 172.21.15.123:0/2026663269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 msgr2=0x557343daa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 0x557343daa570 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5c58011010 tx=0x7f5c580097c0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 -- 172.21.15.123:0/2026663269 shutdown_connections 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99a40 0x557343daa570 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d99050 0x557343d93f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 --2- 172.21.15.123:0/2026663269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d61200 0x557343d93a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 -- 172.21.15.123:0/2026663269 >> 172.21.15.123:0/2026663269 conn(0x557343da36d0 msgr2=0x557343d93330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 -- 172.21.15.123:0/2026663269 shutdown_connections 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.751+0000 7f5c687d3740 1 -- 172.21.15.123:0/2026663269 wait complete. 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 -- start start 2024-03-24T00:45:23.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 Processor -- start 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 0x557343da84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601a9770 con 0x557343d99a40 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601aaa90 con 0x557343d99050 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601a9dd0 con 0x557343d61200 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c687d3740 10 client.0 _ll_register_callbacks cb 0x7f5c500f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c5e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c5e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39064/0 (socket says 172.21.15.123:39064) 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c5e000640 1 -- 172.21.15.123:0/1808983646 learned_addr learned my addr 172.21.15.123:0/1808983646 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c5ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 0x557343da84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.752+0000 7f5c5f400640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 -- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 msgr2=0x557343da89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 -- 172.21.15.123:0/1808983646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 msgr2=0x557343da9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 -- 172.21.15.123:0/1808983646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557343da9f70 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5e000640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c5ea00640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 0x557343da84a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x557343d9a4b0 tx=0x7f5c500f6530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b4bd3a29139aee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5004b040 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5c500b96c0 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c58009e90 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.753+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c50004c60 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 10 client.26431 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557343e0ab80 con 0x557343d61200 2024-03-24T00:45:23.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c58009e90 con 0x557343d61200 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5c500035f0 con 0x557343d61200 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 10 client.26431 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 10 client.26431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 20 client.26431 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343d9ebb0 0x557343e0d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 4 client.26431 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557343d05ae0 con 0x557343d9ebb0 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.754+0000 7f5c5e000640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343d9ebb0 0x557343e0d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c5e000640 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343d9ebb0 0x557343e0d4f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6a3ac6f0ca3afe server_cookie=4a460c1e50742d74 in_seq=0 out_seq=0 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c4cc00640 10 client.26431 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c4cc00640 1 -- 172.21.15.123:0/1808983646 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557343d05ae0 con 0x557343d9ebb0 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c4cc00640 10 client.26431 handle_command_reply: tid=0 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 client.26431 shutdown 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 msgr2=0x557343da84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 0x557343da84a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x557343d9a4b0 tx=0x7f5c500f6530 comp rx=0 tx=0).stop 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 shutdown_connections 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557343d99a40 0x557343da9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557343d99050 0x557343da89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557343d9ebb0 0x557343e0d4f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 --2- 172.21.15.123:0/1808983646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557343d61200 0x557343da84a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c4cc00640 0 client.26431 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 >> 172.21.15.123:0/1808983646 conn(0x557343da36d0 msgr2=0x557343d93bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 shutdown_connections 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.755+0000 7f5c687d3740 1 -- 172.21.15.123:0/1808983646 wait complete. 2024-03-24T00:45:23.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c687d3740 20 client.26431 trim_cache size 0 max 16384 2024-03-24T00:45:23.762 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:45:23.762 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:45:23.762 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:45:23.762 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 msgr2=0x7f5c440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 0x7f5c440a4e60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5c5801f000 tx=0x7f5c5801ed80 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 msgr2=0x7f5c60104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5c5002efd0 tx=0x7f5c5002c660 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 shutdown_connections 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5c440a29a0 0x7f5c440a4e60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c60102030 0x7f5c60102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c60101670 0x7f5c60102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 --2- 172.21.15.123:0/960047559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c60100480 0x7f5c60104410 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.756+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 >> 172.21.15.123:0/960047559 conn(0x7f5c600fba10 msgr2=0x7f5c600fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.757+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 shutdown_connections 2024-03-24T00:45:23.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.757+0000 7f5c5fe00640 1 -- 172.21.15.123:0/960047559 wait complete. 2024-03-24T00:45:23.777 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:45:23.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:23.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 -- 172.21.15.123:0/3530781775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c101510 msgr2=0x7efc7c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 --2- 172.21.15.123:0/3530781775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c101510 0x7efc7c101930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efc70009900 tx=0x7efc7002bc30 comp rx=0 tx=0).stop 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 -- 172.21.15.123:0/3530781775 shutdown_connections 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 --2- 172.21.15.123:0/3530781775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 --2- 172.21.15.123:0/3530781775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c102700 0x7efc7c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 --2- 172.21.15.123:0/3530781775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c101510 0x7efc7c101930 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 -- 172.21.15.123:0/3530781775 >> 172.21.15.123:0/3530781775 conn(0x7efc7c0fca80 msgr2=0x7efc7c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 -- 172.21.15.123:0/3530781775 shutdown_connections 2024-03-24T00:45:23.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.974+0000 7efc82000640 1 -- 172.21.15.123:0/3530781775 wait complete. 2024-03-24T00:45:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.975+0000 7efc82000640 1 Processor -- start 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 -- start start 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 0x7efc7c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c10d1c0 con 0x7efc7c1030c0 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c10d040 con 0x7efc7c101510 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc82000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c10d340 con 0x7efc7c102700 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc80c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc80c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39094/0 (socket says 172.21.15.123:39094) 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc7be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 0x7efc7c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc80c00640 1 -- 172.21.15.123:0/355611239 learned_addr learned my addr 172.21.15.123:0/355611239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.976+0000 7efc81600640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 -- 172.21.15.123:0/355611239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 msgr2=0x7efc7c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 -- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 msgr2=0x7efc7c1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c1a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 -- 172.21.15.123:0/355611239 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c1a1670 con 0x7efc7c102700 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc81600640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc80c00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc7be00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 0x7efc7c199fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e9f0 tx=0x7efc6800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6800cd80 con 0x7efc7c102700 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc680035f0 con 0x7efc7c102700 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68010690 con 0x7efc7c102700 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c1a1960 con 0x7efc7c102700 2024-03-24T00:45:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.977+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c1a1f00 con 0x7efc7c102700 2024-03-24T00:45:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.979+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c1a2190 con 0x7efc7c102700 2024-03-24T00:45:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.981+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efc68003b30 con 0x7efc7c102700 2024-03-24T00:45:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.983+0000 7efc79600640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 0x7efc640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.983+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efc680ced90 con 0x7efc7c102700 2024-03-24T00:45:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.983+0000 7efc80c00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 0x7efc640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.984+0000 7efc80c00640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 0x7efc640a4d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efc70006010 tx=0x7efc7002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:23.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:23.986+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc6809bde0 con 0x7efc7c102700 2024-03-24T00:45:24.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.393+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc7c1a2360 con 0x7efc7c102700 2024-03-24T00:45:24.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.394+0000 7efc79600640 1 -- 172.21.15.123:0/355611239 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7efc6809b6e0 con 0x7efc7c102700 2024-03-24T00:45:24.395 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:45:24.396 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:45:24.396 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 msgr2=0x7efc640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 0x7efc640a4d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efc70006010 tx=0x7efc7002df90 comp rx=0 tx=0).stop 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 msgr2=0x7efc7c199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 0x7efc7c199fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e9f0 tx=0x7efc6800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 shutdown_connections 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efc640a2880 0x7efc640a4d40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7c1030c0 0x7efc7c1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7c102700 0x7efc7c199fe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 --2- 172.21.15.123:0/355611239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc7c101510 0x7efc7c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 >> 172.21.15.123:0/355611239 conn(0x7efc7c0fca80 msgr2=0x7efc7c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 shutdown_connections 2024-03-24T00:45:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.397+0000 7efc82000640 1 -- 172.21.15.123:0/355611239 wait complete. 2024-03-24T00:45:24.418 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:45:24.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 -- 172.21.15.123:0/1663062135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e01015a0 msgr2=0x7f36e01019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 --2- 172.21.15.123:0/1663062135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e01015a0 0x7f36e01019c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009900 tx=0x7f36d402bc30 comp rx=0 tx=0).stop 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 -- 172.21.15.123:0/1663062135 shutdown_connections 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 --2- 172.21.15.123:0/1663062135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e0103150 0x7f36e01099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 --2- 172.21.15.123:0/1663062135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 0x7f36e0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 --2- 172.21.15.123:0/1663062135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e01015a0 0x7f36e01019c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 -- 172.21.15.123:0/1663062135 >> 172.21.15.123:0/1663062135 conn(0x7f36e00fcb10 msgr2=0x7f36e00fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 -- 172.21.15.123:0/1663062135 shutdown_connections 2024-03-24T00:45:24.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.609+0000 7f36e6a00640 1 -- 172.21.15.123:0/1663062135 wait complete. 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 Processor -- start 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 -- start start 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e01015a0 0x7f36e0199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 0x7f36e019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010d240 con 0x7f36e01015a0 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010d0c0 con 0x7f36e0103150 2024-03-24T00:45:24.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010d3c0 con 0x7f36e0102790 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e01015a0 0x7f36e0199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39112/0 (socket says 172.21.15.123:39112) 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e6000640 1 -- 172.21.15.123:0/2319752392 learned_addr learned my addr 172.21.15.123:0/2319752392 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.611+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 0x7f36e019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 -- 172.21.15.123:0/2319752392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 msgr2=0x7f36e019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 0x7f36e019a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 -- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e01015a0 msgr2=0x7f36e0199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e01015a0 0x7f36e0199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 -- 172.21.15.123:0/2319752392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e01a1740 con 0x7f36e0103150 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6000640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f36d800eb10 tx=0x7f36d800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36de000640 1 -- 172.21.15.123:0/2319752392 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8004b20 con 0x7f36e0103150 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36de000640 1 -- 172.21.15.123:0/2319752392 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36d8004cc0 con 0x7f36e0103150 2024-03-24T00:45:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e01a1a30 con 0x7f36e0103150 2024-03-24T00:45:24.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36de000640 1 -- 172.21.15.123:0/2319752392 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8010760 con 0x7f36e0103150 2024-03-24T00:45:24.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.612+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e01a1f30 con 0x7f36e0103150 2024-03-24T00:45:24.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36de000640 1 -- 172.21.15.123:0/2319752392 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f36d8005020 con 0x7f36e0103150 2024-03-24T00:45:24.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 Processor -- start 2024-03-24T00:45:24.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 -- start start 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a13e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a1be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.616+0000 7f36ead99740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb779782c0 con 0x55cb77a139f0 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36ead99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb77920c90 con 0x55cb77a14bc0 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36ead99740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb77974360 con 0x55cb77a15580 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e4c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a13e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39124/0 (socket says 172.21.15.123:39124) 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 -- 172.21.15.123:0/703595010 learned_addr learned my addr 172.21.15.123:0/703595010 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e4c00640 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a1be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 -- 172.21.15.123:0/703595010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 msgr2=0x55cb77a1be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a1be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 -- 172.21.15.123:0/703595010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 msgr2=0x55cb77a13e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a13e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 -- 172.21.15.123:0/703595010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb77945080 con 0x55cb77a14bc0 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e4c00640 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a13e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.617+0000 7f36e5600640 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000c00 tx=0x7f36d4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4c76c192df9c708 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36de000640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 0x7f36c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36d2600640 1 -- 172.21.15.123:0/703595010 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d402c980 con 0x55cb77a14bc0 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36d2600640 1 -- 172.21.15.123:0/703595010 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36d402cb20 con 0x55cb77a14bc0 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36d2600640 1 -- 172.21.15.123:0/703595010 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d40046c0 con 0x55cb77a14bc0 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36de000640 1 -- 172.21.15.123:0/2319752392 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f36d80cefe0 con 0x7f36e0103150 2024-03-24T00:45:24.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.618+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 0x7f36c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- 172.21.15.123:0/703595010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 msgr2=0x55cb77a15040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000c00 tx=0x7f36d4009560 comp rx=0 tx=0).stop 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 0x7f36c80a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55cb77a14ac0 tx=0x7f36cc015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- 172.21.15.123:0/703595010 shutdown_connections 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a1be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a15040 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 --2- 172.21.15.123:0/703595010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a13e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- 172.21.15.123:0/703595010 >> 172.21.15.123:0/703595010 conn(0x55cb77a0ede0 msgr2=0x55cb77a0f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- 172.21.15.123:0/703595010 shutdown_connections 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- 172.21.15.123:0/703595010 wait complete. 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 -- start start 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.619+0000 7f36ead99740 1 Processor -- start 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a26380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a2d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb77a1c530 con 0x55cb77a139f0 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb779340d0 con 0x55cb77a14bc0 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb77a1c6b0 con 0x55cb77a15580 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36ead99740 10 client.0 _ll_register_callbacks cb 0x55cb77938e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36e5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a26380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36e4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36e4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39136/0 (socket says 172.21.15.123:39136) 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36e4c00640 1 -- 172.21.15.123:0/1438415484 learned_addr learned my addr 172.21.15.123:0/1438415484 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:24.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.620+0000 7f36e6000640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a2d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 -- 172.21.15.123:0/1438415484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 msgr2=0x55cb77a2d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a2d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 -- 172.21.15.123:0/1438415484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 msgr2=0x55cb77a26380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a26380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 -- 172.21.15.123:0/1438415484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb7768c3f0 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0097c0 tx=0x7f36cc009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff5d54bc135acc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc019030 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36d402cb20 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.621+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f36cc003c40 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36cc003e40 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 10 client.26438 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cb777567d0 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36d402cb20 con 0x55cb77a14bc0 2024-03-24T00:45:24.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f36cc020c00 con 0x55cb77a14bc0 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 10 client.26438 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 10 client.26438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 20 client.26438 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a31e00 0x55cb77a34280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 4 client.26438 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.622+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb77985800 con 0x55cb77a31e00 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.623+0000 7f36e5600640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a31e00 0x55cb77a34280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.623+0000 7f36e5600640 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a31e00 0x55cb77a34280 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1242340ba7c54234 server_cookie=c3ef17f6fc63abde in_seq=0 out_seq=0 2024-03-24T00:45:24.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.623+0000 7f36c6a00640 10 client.26438 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:24.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.624+0000 7f36c6a00640 1 -- 172.21.15.123:0/1438415484 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cb77985800 con 0x55cb77a31e00 2024-03-24T00:45:24.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.624+0000 7f36c6a00640 10 client.26438 handle_command_reply: tid=0 2024-03-24T00:45:24.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.624+0000 7f36ead99740 1 client.26438 shutdown 2024-03-24T00:45:24.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.624+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 msgr2=0x55cb77a268c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.624+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0097c0 tx=0x7f36cc009370 comp rx=0 tx=0).stop 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 shutdown_connections 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a31e00 0x55cb77a34280 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a15580 0x55cb77a2d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a14bc0 0x55cb77a268c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 --2- 172.21.15.123:0/1438415484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a139f0 0x55cb77a26380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36c6a00640 0 client.26438 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 >> 172.21.15.123:0/1438415484 conn(0x55cb77a0ede0 msgr2=0x55cb77a0e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 shutdown_connections 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 1 -- 172.21.15.123:0/1438415484 wait complete. 2024-03-24T00:45:24.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.625+0000 7f36ead99740 20 client.26438 trim_cache size 0 max 16384 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 Processor -- start 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 -- start start 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a17df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a47c0 con 0x55cb77a15580 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a6d40 con 0x55cb77a139f0 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.640+0000 7f36ead99740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a4dc0 con 0x55cb77a14bc0 2024-03-24T00:45:24.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50270/0 (socket says 172.21.15.123:50270) 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39152/0 (socket says 172.21.15.123:39152) 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 -- 172.21.15.123:0/2376339353 learned_addr learned my addr 172.21.15.123:0/2376339353 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e6000640 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a17df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 -- 172.21.15.123:0/2376339353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 msgr2=0x55cb77a1b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 -- 172.21.15.123:0/2376339353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 msgr2=0x55cb77a17df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a17df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 -- 172.21.15.123:0/2376339353 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb77a1b640 con 0x55cb77a14bc0 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e5600640 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36e4c00640 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc021510 tx=0x7f36cc005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e78d82edd8eb20cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36d1c00640 1 -- 172.21.15.123:0/2376339353 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc03a070 con 0x55cb77a14bc0 2024-03-24T00:45:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.641+0000 7f36d1c00640 1 -- 172.21.15.123:0/2376339353 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc01b050 con 0x55cb77a14bc0 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36d1c00640 1 -- 172.21.15.123:0/2376339353 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc021e50 con 0x55cb77a14bc0 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 -- 172.21.15.123:0/2376339353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 msgr2=0x55cb77a2d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc021510 tx=0x7f36cc005e30 comp rx=0 tx=0).stop 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 -- 172.21.15.123:0/2376339353 shutdown_connections 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2d8c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 --2- 172.21.15.123:0/2376339353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a1b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 -- 172.21.15.123:0/2376339353 >> 172.21.15.123:0/2376339353 conn(0x55cb77a0ede0 msgr2=0x55cb77a128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 -- 172.21.15.123:0/2376339353 shutdown_connections 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.642+0000 7f36ead99740 1 -- 172.21.15.123:0/2376339353 wait complete. 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 -- start start 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 Processor -- start 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a2c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a53c0 con 0x55cb77a15580 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a66e0 con 0x55cb77a139f0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01a5a20 con 0x55cb77a14bc0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.643+0000 7f36ead99740 10 client.0 _ll_register_callbacks cb 0x7f36cc009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39156/0 (socket says 172.21.15.123:39156) 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 -- 172.21.15.123:0/1702718908 learned_addr learned my addr 172.21.15.123:0/1702718908 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e6000640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a2c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 -- 172.21.15.123:0/1702718908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 msgr2=0x55cb77a2bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 -- 172.21.15.123:0/1702718908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 msgr2=0x55cb77a2c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a2c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 -- 172.21.15.123:0/1702718908 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb77a2cb10 con 0x55cb77a139f0 2024-03-24T00:45:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36e5600640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36d4005c40 tx=0x7f36d4005a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=323e98308df9865b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4055060 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f36d4005240 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.644+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36cc01b050 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36d4005480 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 10 client.26444 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cb77a8e890 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc01b050 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f36d403f560 con 0x55cb77a139f0 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 10 client.26444 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 10 client.26444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 20 client.26444 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a8ee30 0x55cb77a91250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 4 client.26444 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cb7797fb50 con 0x55cb77a8ee30 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.645+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a8ee30 0x55cb77a91250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36e4c00640 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a8ee30 0x55cb77a91250 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407696e4517f92a0 server_cookie=32f91f0236e1f5cf in_seq=0 out_seq=0 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36c6a00640 10 client.26444 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36c6a00640 1 -- 172.21.15.123:0/1702718908 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cb7797fb50 con 0x55cb77a8ee30 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36c6a00640 10 client.26444 handle_command_reply: tid=0 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 client.26444 shutdown 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 msgr2=0x55cb77a2d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36d4005c40 tx=0x7f36d4005a60 comp rx=0 tx=0).stop 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 shutdown_connections 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cb77a15580 0x55cb77a2c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55cb77a8ee30 0x55cb77a91250 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36c6a00640 0 client.26444 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cb77a14bc0 0x55cb77a2bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 --2- 172.21.15.123:0/1702718908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb77a139f0 0x55cb77a2d7b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 >> 172.21.15.123:0/1702718908 conn(0x55cb77a0ede0 msgr2=0x55cb77a128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.646+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 shutdown_connections 2024-03-24T00:45:24.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36ead99740 1 -- 172.21.15.123:0/1702718908 wait complete. 2024-03-24T00:45:24.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36ead99740 20 client.26444 trim_cache size 0 max 16384 2024-03-24T00:45:24.652 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 msgr2=0x7f36c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 0x7f36c80a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55cb77a14ac0 tx=0x7f36cc015040 comp rx=0 tx=0).stop 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 msgr2=0x7f36e01a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f36d800eb10 tx=0x7f36d800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 shutdown_connections 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e0103150 0x7f36e01a10d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36e0102790 0x7f36e019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f36c80a2800 0x7f36c80a4cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 --2- 172.21.15.123:0/2319752392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e01015a0 0x7f36e0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.647+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 >> 172.21.15.123:0/2319752392 conn(0x7f36e00fcb10 msgr2=0x7f36e00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:24.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.648+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 shutdown_connections 2024-03-24T00:45:24.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:24.648+0000 7f36e6a00640 1 -- 172.21.15.123:0/2319752392 wait complete. 2024-03-24T00:45:24.662 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:24 smithi123 ceph-mon[31487]: pgmap v610: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:45:24.662 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:24 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/355611239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:45:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:24 smithi177 ceph-mon[36034]: pgmap v610: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:45:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:24 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/355611239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:45:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:24 smithi204 ceph-mon[36345]: pgmap v610: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 37 MiB/s wr, 3.33k op/s 2024-03-24T00:45:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:24 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/355611239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:45:25.665 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:45:25.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:25.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:45:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 -- 172.21.15.123:0/3972859089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 msgr2=0x7f49e4103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3972859089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4103be0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009900 tx=0x7f49d402bc30 comp rx=0 tx=0).stop 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 -- 172.21.15.123:0/3972859089 shutdown_connections 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3972859089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e4105370 0x7f49e410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3972859089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e41049b0 0x7f49e4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3972859089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4103be0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 -- 172.21.15.123:0/3972859089 >> 172.21.15.123:0/3972859089 conn(0x7f49e40fed50 msgr2=0x7f49e4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 -- 172.21.15.123:0/3972859089 shutdown_connections 2024-03-24T00:45:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.888+0000 7f49e3e00640 1 -- 172.21.15.123:0/3972859089 wait complete. 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.889+0000 7f49e3e00640 1 Processor -- start 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.889+0000 7f49e3e00640 1 -- start start 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e41049b0 0x7f49e419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e4105370 0x7f49e41a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e410f460 con 0x7f49e4105370 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e410f2e0 con 0x7f49e41037c0 2024-03-24T00:45:25.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e410f5e0 con 0x7f49e41049b0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53524/0 (socket says 172.21.15.123:53524) 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 -- 172.21.15.123:0/3581256782 learned_addr learned my addr 172.21.15.123:0/3581256782 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2000640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e41049b0 0x7f49e419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e3400640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e4105370 0x7f49e41a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 -- 172.21.15.123:0/3581256782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e41049b0 msgr2=0x7f49e419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e41049b0 0x7f49e419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 -- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e4105370 msgr2=0x7f49e41a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e4105370 0x7f49e41a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.890+0000 7f49e2a00640 1 -- 172.21.15.123:0/3581256782 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e41a1730 con 0x7f49e41037c0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49e2a00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f49d4000c00 tx=0x7f49d402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49db400640 1 -- 172.21.15.123:0/3581256782 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4043020 con 0x7f49e41037c0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49db400640 1 -- 172.21.15.123:0/3581256782 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d40054e0 con 0x7f49e41037c0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e41a19c0 con 0x7f49e41037c0 2024-03-24T00:45:25.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49db400640 1 -- 172.21.15.123:0/3581256782 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4003bd0 con 0x7f49e41037c0 2024-03-24T00:45:25.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.891+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e41a1f60 con 0x7f49e41037c0 2024-03-24T00:45:25.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.894+0000 7f49db400640 1 -- 172.21.15.123:0/3581256782 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f49d4046020 con 0x7f49e41037c0 2024-03-24T00:45:25.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.894+0000 7f49ec3a6740 1 Processor -- start 2024-03-24T00:45:25.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.894+0000 7f49ec3a6740 1 -- start start 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8ef67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8ef92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8e56eb0 con 0x55dba8ef6e10 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8dffa60 con 0x55dba8ef12b0 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8e52f50 con 0x55dba8ef1ca0 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8ef67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53538/0 (socket says 172.21.15.123:53538) 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e3400640 1 -- 172.21.15.123:0/2135712812 learned_addr learned my addr 172.21.15.123:0/2135712812 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2000640 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8ef92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 -- 172.21.15.123:0/2135712812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 msgr2=0x55dba8ef16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 -- 172.21.15.123:0/2135712812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 msgr2=0x55dba8ef92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8ef92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.895+0000 7f49e2a00640 1 -- 172.21.15.123:0/2135712812 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dba8c337e0 con 0x55dba8ef1ca0 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8ef67a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ee4a0 tx=0x7f49d40f6ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54476412e4da038a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49db400640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 0x7f49c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49cf400640 1 -- 172.21.15.123:0/2135712812 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4100020 con 0x55dba8ef1ca0 2024-03-24T00:45:25.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49cf400640 1 -- 172.21.15.123:0/2135712812 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d40bda70 con 0x55dba8ef1ca0 2024-03-24T00:45:25.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49cf400640 1 -- 172.21.15.123:0/2135712812 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d40d9e50 con 0x55dba8ef1ca0 2024-03-24T00:45:25.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49e3400640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 0x7f49c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.896+0000 7f49db400640 1 -- 172.21.15.123:0/3581256782 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49d40f09c0 con 0x7f49e41037c0 2024-03-24T00:45:25.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2135712812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 msgr2=0x55dba8ef67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8ef67a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ee4a0 tx=0x7f49d40f6ae0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2135712812 shutdown_connections 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8ef92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8ef67a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2135712812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2135712812 >> 172.21.15.123:0/2135712812 conn(0x55dba8eec820 msgr2=0x55dba8eecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2135712812 shutdown_connections 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49e3400640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 0x7f49c80a4d40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x55dba8ef2500 tx=0x7f49dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.897+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2135712812 wait complete. 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 -- start start 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 Processor -- start 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8efb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8efb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8ef9b80 con 0x55dba8ef1ca0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8e12eb0 con 0x55dba8ef12b0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba8ef9d00 con 0x55dba8ef6e10 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49ec3a6740 10 client.0 _ll_register_callbacks cb 0x55dba8e07150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49e3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8efb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49e3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54712/0 (socket says 172.21.15.123:54712) 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.898+0000 7f49e3400640 1 -- 172.21.15.123:0/344680712 learned_addr learned my addr 172.21.15.123:0/344680712 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e2000640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8efb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 -- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 msgr2=0x55dba8efb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8efb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 -- 172.21.15.123:0/344680712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 msgr2=0x55dba8efb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8efb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 -- 172.21.15.123:0/344680712 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dba8c93f70 con 0x55dba8ef6e10 2024-03-24T00:45:25.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49e3400640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00fb90 tx=0x7f49dc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea535790e1ea4569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00fe00 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49dc018030 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.899+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49d40bda70 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49dc003970 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 10 client.26467 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55dba8b2b170 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49d40bda70 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f49dc004290 con 0x55dba8ef6e10 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 10 client.26467 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 10 client.26467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 20 client.26467 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f07080 0x55dba8f094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 4 client.26467 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:25.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.900+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dba8e643f0 con 0x55dba8f07080 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.901+0000 7f49e2a00640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f07080 0x55dba8f094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.901+0000 7f49e2a00640 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f07080 0x55dba8f094e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef59cf5aa515074f server_cookie=780620365442f69d in_seq=0 out_seq=0 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.901+0000 7f49c3e00640 10 client.26467 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49c3e00640 1 -- 172.21.15.123:0/344680712 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dba8e643f0 con 0x55dba8f07080 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49c3e00640 10 client.26467 handle_command_reply: tid=0 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 client.26467 shutdown 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 msgr2=0x55dba8f02730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00fb90 tx=0x7f49dc004ab0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 shutdown_connections 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f07080 0x55dba8f094e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f02730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8efb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49c3e00640 0 client.26467 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/344680712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8efb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 >> 172.21.15.123:0/344680712 conn(0x55dba8eec820 msgr2=0x55dba8eeba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 shutdown_connections 2024-03-24T00:45:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 1 -- 172.21.15.123:0/344680712 wait complete. 2024-03-24T00:45:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.902+0000 7f49ec3a6740 20 client.26467 trim_cache size 0 max 16384 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 Processor -- start 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 -- start start 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8f026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41a8b40 con 0x55dba8ef1ca0 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41ab100 con 0x55dba8ef12b0 2024-03-24T00:45:25.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41a9140 con 0x55dba8ef6e10 2024-03-24T00:45:25.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8f026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54716/0 (socket says 172.21.15.123:54716) 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e3400640 1 -- 172.21.15.123:0/1909371172 learned_addr learned my addr 172.21.15.123:0/1909371172 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.921+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53552/0 (socket says 172.21.15.123:53552) 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 -- 172.21.15.123:0/1909371172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 msgr2=0x55dba8ef00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 -- 172.21.15.123:0/1909371172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 msgr2=0x55dba8f026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8f026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 -- 172.21.15.123:0/1909371172 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dba8eed290 con 0x55dba8ef6e10 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e2a00640 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49e3400640 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f49dc019040 tx=0x7f49dc009fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f55f7054441583c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49cea00640 1 -- 172.21.15.123:0/1909371172 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc038040 con 0x55dba8ef6e10 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49cea00640 1 -- 172.21.15.123:0/1909371172 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49dc01ecd0 con 0x55dba8ef6e10 2024-03-24T00:45:25.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.922+0000 7f49cea00640 1 -- 172.21.15.123:0/1909371172 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc003f70 con 0x55dba8ef6e10 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 -- 172.21.15.123:0/1909371172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 msgr2=0x55dba8f0dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f49dc019040 tx=0x7f49dc009fa0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 -- 172.21.15.123:0/1909371172 shutdown_connections 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef6e10 0x55dba8f0dda0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef1ca0 0x55dba8f026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/1909371172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8ef00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 -- 172.21.15.123:0/1909371172 >> 172.21.15.123:0/1909371172 conn(0x55dba8eec820 msgr2=0x55dba8eeba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 -- 172.21.15.123:0/1909371172 shutdown_connections 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.923+0000 7f49ec3a6740 1 -- 172.21.15.123:0/1909371172 wait complete. 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 -- start start 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 Processor -- start 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8f01310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8f01850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41a9780 con 0x55dba8ef6e10 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41aaaa0 con 0x55dba8ef12b0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41a9de0 con 0x55dba8ef1ca0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49ec3a6740 10 client.0 _ll_register_callbacks cb 0x7f49dc009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53560/0 (socket says 172.21.15.123:53560) 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49e2a00640 1 -- 172.21.15.123:0/2261117716 learned_addr learned my addr 172.21.15.123:0/2261117716 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.924+0000 7f49e3400640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8f01850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2000640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8f01310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 -- 172.21.15.123:0/2261117716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 msgr2=0x55dba8f01310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8f01310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 -- 172.21.15.123:0/2261117716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 msgr2=0x55dba8f01850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8f01850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 -- 172.21.15.123:0/2261117716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dba8f62970 con 0x55dba8ef12b0 2024-03-24T00:45:25.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e3400640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8f01850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49e2a00640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f49d40b97d0 tx=0x7f49d4005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4afa550bc63a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d40dd2a0 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49d40f6230 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.925+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49dc01ecd0 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49d40dd5f0 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 10 client.26462 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55dba8f63b30 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49dc01ecd0 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f49d40b99e0 con 0x55dba8ef12b0 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 10 client.26462 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 10 client.26462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 20 client.26462 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:25.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f642f0 0x55dba8f66710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:25.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 4 client.26462 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:45:25.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55dba8e5e740 con 0x55dba8f642f0 2024-03-24T00:45:25.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.926+0000 7f49e2000640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f642f0 0x55dba8f66710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:25.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.927+0000 7f49e2000640 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f642f0 0x55dba8f66710 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fc6a6bcb6c0006a server_cookie=de476f4b4e9d42d1 in_seq=0 out_seq=0 2024-03-24T00:45:25.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.927+0000 7f49c3e00640 10 client.26462 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49c3e00640 1 -- 172.21.15.123:0/2261117716 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55dba8e5e740 con 0x55dba8f642f0 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49c3e00640 10 client.26462 handle_command_reply: tid=0 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 client.26462 shutdown 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 msgr2=0x55dba8f00dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f49d40b97d0 tx=0x7f49d4005dd0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 shutdown_connections 2024-03-24T00:45:25.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dba8ef6e10 0x55dba8f01850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55dba8f642f0 0x55dba8f66710 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dba8ef1ca0 0x55dba8f01310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 --2- 172.21.15.123:0/2261117716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dba8ef12b0 0x55dba8f00dd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49c3e00640 0 client.26462 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 >> 172.21.15.123:0/2261117716 conn(0x55dba8eec820 msgr2=0x55dba8ef2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 shutdown_connections 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.928+0000 7f49ec3a6740 1 -- 172.21.15.123:0/2261117716 wait complete. 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.929+0000 7f49ec3a6740 20 client.26462 trim_cache size 0 max 16384 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:45:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "762d32e1-29f5-4dbd-8d66-da389d05b26e", 2024-03-24T00:45:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:45:25.934 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:45:25.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.929+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 msgr2=0x7f49c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.929+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 0x7f49c80a4d40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x55dba8ef2500 tx=0x7f49dc009500 comp rx=0 tx=0).stop 2024-03-24T00:45:25.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.929+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 msgr2=0x7f49e4199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.929+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f49d4000c00 tx=0x7f49d402c650 comp rx=0 tx=0).stop 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 shutdown_connections 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f49c80a2880 0x7f49c80a4d40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49e4105370 0x7f49e41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49e41049b0 0x7f49e419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 --2- 172.21.15.123:0/3581256782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e41037c0 0x7f49e4199b10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 >> 172.21.15.123:0/3581256782 conn(0x7f49e40fed50 msgr2=0x7f49e4107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 shutdown_connections 2024-03-24T00:45:25.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:25.930+0000 7f49e3e00640 1 -- 172.21.15.123:0/3581256782 wait complete. 2024-03-24T00:45:25.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:45:25.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:45:26.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.222+0000 7f6aab000640 1 -- 172.21.15.123:0/3069615236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41028c0 msgr2=0x7f6aa4102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.222+0000 7f6aab000640 1 --2- 172.21.15.123:0/3069615236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41028c0 0x7f6aa4102d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009900 tx=0x7f6a9002bc50 comp rx=0 tx=0).stop 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.222+0000 7f6aab000640 1 -- 172.21.15.123:0/3069615236 shutdown_connections 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.222+0000 7f6aab000640 1 --2- 172.21.15.123:0/3069615236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4103280 0x7f6aa4109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.223+0000 7f6aab000640 1 --2- 172.21.15.123:0/3069615236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41028c0 0x7f6aa4102d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.223+0000 7f6aab000640 1 --2- 172.21.15.123:0/3069615236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa41016d0 0x7f6aa4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.223+0000 7f6aab000640 1 -- 172.21.15.123:0/3069615236 >> 172.21.15.123:0/3069615236 conn(0x7f6aa40fcc40 msgr2=0x7f6aa40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.223+0000 7f6aab000640 1 -- 172.21.15.123:0/3069615236 shutdown_connections 2024-03-24T00:45:26.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.223+0000 7f6aab000640 1 -- 172.21.15.123:0/3069615236 wait complete. 2024-03-24T00:45:26.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.225+0000 7f6aab000640 1 Processor -- start 2024-03-24T00:45:26.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.225+0000 7f6aab000640 1 -- start start 2024-03-24T00:45:26.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.227+0000 7f6aab000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 0x7f6aa4195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.227+0000 7f6aab000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa41028c0 0x7f6aa4195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.227+0000 7f6aab000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.228+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 0x7f6aa4195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.228+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 0x7f6aa4195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54744/0 (socket says 172.21.15.123:54744) 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aa9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aa9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53578/0 (socket says 172.21.15.123:53578) 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aa9000640 1 -- 172.21.15.123:0/4034973664 learned_addr learned my addr 172.21.15.123:0/4034973664 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa410d350 con 0x7f6aa41028c0 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa410d1d0 con 0x7f6aa4103280 2024-03-24T00:45:26.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.232+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa410d4d0 con 0x7f6aa41016d0 2024-03-24T00:45:26.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.233+0000 7f6aa9000640 1 -- 172.21.15.123:0/4034973664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 msgr2=0x7f6aa4195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.233+0000 7f6aa9000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 0x7f6aa4195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.233+0000 7f6aa9000640 1 -- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa41028c0 msgr2=0x7f6aa4195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.233+0000 7f6aa9000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa41028c0 0x7f6aa4195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.233+0000 7f6aa9000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa419aeb0 con 0x7f6aa4103280 2024-03-24T00:45:26.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.234+0000 7f6aa9000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00cce0 tx=0x7f6a9c00a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.234+0000 7f6aa0c00640 1 -- 172.21.15.123:0/4034973664 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c00f450 con 0x7f6aa4103280 2024-03-24T00:45:26.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.234+0000 7f6aa0c00640 1 -- 172.21.15.123:0/4034973664 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a9c00fad0 con 0x7f6aa4103280 2024-03-24T00:45:26.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.234+0000 7f6aa0c00640 1 -- 172.21.15.123:0/4034973664 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c018c10 con 0x7f6aa4103280 2024-03-24T00:45:26.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.235+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa41ae7a0 con 0x7f6aa4103280 2024-03-24T00:45:26.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.235+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa41aec10 con 0x7f6aa4103280 2024-03-24T00:45:26.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.246+0000 7f6aa0c00640 1 -- 172.21.15.123:0/4034973664 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6a9c0213f0 con 0x7f6aa4103280 2024-03-24T00:45:26.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.251+0000 7f6aadd9f740 1 Processor -- start 2024-03-24T00:45:26.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.251+0000 7f6aadd9f740 1 -- start start 2024-03-24T00:45:26.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.252+0000 7f6aa0c00640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 0x7f6a880a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.252+0000 7f6aa0c00640 1 -- 172.21.15.123:0/4034973664 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6a9c00f5f0 con 0x7f6aa4103280 2024-03-24T00:45:26.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.254+0000 7f6aa3400640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 0x7f6a880a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.255+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.256+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74e870 0x563ffe753370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.256+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe755e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.256+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.256+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54746/0 (socket says 172.21.15.123:54746) 2024-03-24T00:45:26.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.257+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe6b3250 con 0x563ffe7539e0 2024-03-24T00:45:26.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.257+0000 7f6aa3e00640 1 -- 172.21.15.123:0/4249848957 learned_addr learned my addr 172.21.15.123:0/4249848957 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:26.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.257+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe65ba30 con 0x563ffe74e870 2024-03-24T00:45:26.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.257+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe6af2f0 con 0x563ffe74de80 2024-03-24T00:45:26.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.257+0000 7f6aa9000640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe755e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.258+0000 7f6aa3400640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74e870 0x563ffe753370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa3e00640 1 -- 172.21.15.123:0/4249848957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74e870 msgr2=0x563ffe753370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74e870 0x563ffe753370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa3e00640 1 -- 172.21.15.123:0/4249848957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 msgr2=0x563ffe755e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe755e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa3e00640 1 -- 172.21.15.123:0/4249848957 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ffe646690 con 0x563ffe74de80 2024-03-24T00:45:26.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.259+0000 7f6aa9000640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe755e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:26.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.260+0000 7f6aa3400640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 0x7f6a880a4dc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6a90037040 tx=0x7f6a90000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.260+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009ca0 tx=0x7f6a94011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97608e7d2ad5cf99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.261+0000 7f6a99600640 1 -- 172.21.15.123:0/4249848957 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94004220 con 0x563ffe74de80 2024-03-24T00:45:26.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.262+0000 7f6a99600640 1 -- 172.21.15.123:0/4249848957 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a94009e50 con 0x563ffe74de80 2024-03-24T00:45:26.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.262+0000 7f6a99600640 1 -- 172.21.15.123:0/4249848957 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a940197a0 con 0x563ffe74de80 2024-03-24T00:45:26.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.263+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 msgr2=0x563ffe74e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.263+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009ca0 tx=0x7f6a94011810 comp rx=0 tx=0).stop 2024-03-24T00:45:26.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.265+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 shutdown_connections 2024-03-24T00:45:26.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.265+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe755e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.265+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74e870 0x563ffe753370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.265+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/4249848957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74de80 0x563ffe74e2a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.265+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 >> 172.21.15.123:0/4249848957 conn(0x563ffe716200 msgr2=0x563ffe7497f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.266+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 shutdown_connections 2024-03-24T00:45:26.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.266+0000 7f6aadd9f740 1 -- 172.21.15.123:0/4249848957 wait complete. 2024-03-24T00:45:26.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.266+0000 7f6aadd9f740 1 -- start start 2024-03-24T00:45:26.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.267+0000 7f6aadd9f740 1 Processor -- start 2024-03-24T00:45:26.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.270+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe757e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.271+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe7583c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.271+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe75f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.271+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe757e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.271+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe756990 con 0x563ffe7539e0 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.271+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe66ee80 con 0x563ffe74de80 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aa3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe7583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aa3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe7583c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54756/0 (socket says 172.21.15.123:54756) 2024-03-24T00:45:26.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aa3400640 1 -- 172.21.15.123:0/1818214053 learned_addr learned my addr 172.21.15.123:0/1818214053 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:26.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ffe756b10 con 0x563ffe74e870 2024-03-24T00:45:26.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aadd9f740 10 client.0 _ll_register_callbacks cb 0x563ffe681130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:26.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.272+0000 7f6aa9000640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe75f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.275+0000 7f6aa9000640 1 -- 172.21.15.123:0/1818214053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 msgr2=0x563ffe7583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.275+0000 7f6aa9000640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe7583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.275+0000 7f6aa9000640 1 -- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 msgr2=0x563ffe757e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.275+0000 7f6aa9000640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe757e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.275+0000 7f6aa9000640 1 -- 172.21.15.123:0/1818214053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ffe4917d0 con 0x563ffe7539e0 2024-03-24T00:45:26.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.276+0000 7f6aa9000640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe75f400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c099460 tx=0x7f6a9c09d250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb147efe311fdd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.276+0000 7f6a8e000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:45:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.277+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0a5230 con 0x563ffe7539e0 2024-03-24T00:45:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.278+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6a9c0a4da0 con 0x563ffe7539e0 2024-03-24T00:45:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.281+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a94009e50 con 0x563ffe7539e0 2024-03-24T00:45:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.281+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a9c0a4fc0 con 0x563ffe7539e0 2024-03-24T00:45:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.281+0000 7f6aadd9f740 10 client.16302 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.281+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563ffe3c73f0 con 0x563ffe7539e0 2024-03-24T00:45:26.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.282+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a94009e50 con 0x563ffe7539e0 2024-03-24T00:45:26.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6a9c0987e0 con 0x563ffe7539e0 2024-03-24T00:45:26.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 10 client.16302 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:26.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 10 client.16302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:26.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 20 client.16302 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:26.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe763bd0 0x563ffe766030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 4 client.16302 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:26.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ffe6c0790 con 0x563ffe763bd0 2024-03-24T00:45:26.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.283+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe763bd0 0x563ffe766030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.285+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe763bd0 0x563ffe766030 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1301aabb5902a9a server_cookie=93d77aff803caf8c in_seq=0 out_seq=0 2024-03-24T00:45:26.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.285+0000 7f6a8e000640 10 client.16302 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:26.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.286+0000 7f6a8e000640 1 -- 172.21.15.123:0/1818214053 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563ffe6c0790 con 0x563ffe763bd0 2024-03-24T00:45:26.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.287+0000 7f6a8e000640 10 client.16302 handle_command_reply: tid=0 2024-03-24T00:45:26.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.287+0000 7f6aadd9f740 1 client.16302 shutdown 2024-03-24T00:45:26.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.288+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 msgr2=0x563ffe75f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.288+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe75f400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c099460 tx=0x7f6a9c09d250 comp rx=0 tx=0).stop 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 shutdown_connections 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe7539e0 0x563ffe75f400 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe763bd0 0x563ffe766030 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe7583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/1818214053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe757e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6a8e000640 0 client.16302 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 >> 172.21.15.123:0/1818214053 conn(0x563ffe716200 msgr2=0x563ffe748a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.289+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 shutdown_connections 2024-03-24T00:45:26.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.290+0000 7f6aadd9f740 1 -- 172.21.15.123:0/1818214053 wait complete. 2024-03-24T00:45:26.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.290+0000 7f6aadd9f740 20 client.16302 trim_cache size 0 max 16384 2024-03-24T00:45:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.352+0000 7f6aadd9f740 1 Processor -- start 2024-03-24T00:45:26.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.353+0000 7f6aadd9f740 1 -- start start 2024-03-24T00:45:26.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74e870 0x563ffe75f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe7539e0 0x563ffe76a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b5910 con 0x563ffe74e870 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b7ed0 con 0x563ffe74de80 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b5f10 con 0x563ffe7539e0 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aa3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53604/0 (socket says 172.21.15.123:53604) 2024-03-24T00:45:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.355+0000 7f6aa3e00640 1 -- 172.21.15.123:0/3823699871 learned_addr learned my addr 172.21.15.123:0/3823699871 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.356+0000 7f6aa3e00640 1 -- 172.21.15.123:0/3823699871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe7539e0 msgr2=0x563ffe76a940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:45:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.356+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe7539e0 0x563ffe76a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.356+0000 7f6aa3e00640 1 -- 172.21.15.123:0/3823699871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74e870 msgr2=0x563ffe75f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.356+0000 7f6aa3400640 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74e870 0x563ffe75f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.358+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74e870 0x563ffe75f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.358+0000 7f6aa3e00640 1 -- 172.21.15.123:0/3823699871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ffe3870f0 con 0x563ffe74de80 2024-03-24T00:45:26.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.359+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6a94011d10 tx=0x7f6a940096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b673b2cbf36d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.359+0000 7f6a98c00640 1 -- 172.21.15.123:0/3823699871 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94009d90 con 0x563ffe74de80 2024-03-24T00:45:26.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.361+0000 7f6a98c00640 1 -- 172.21.15.123:0/3823699871 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a94009780 con 0x563ffe74de80 2024-03-24T00:45:26.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.361+0000 7f6a98c00640 1 -- 172.21.15.123:0/3823699871 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94009d90 con 0x563ffe74de80 2024-03-24T00:45:26.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.362+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3823699871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 msgr2=0x563ffe748910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.362+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6a94011d10 tx=0x7f6a940096a0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.363+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3823699871 shutdown_connections 2024-03-24T00:45:26.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.363+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe7539e0 0x563ffe76a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.363+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74e870 0x563ffe75f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.363+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3823699871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe74de80 0x563ffe748910 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.363+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3823699871 >> 172.21.15.123:0/3823699871 conn(0x563ffe716200 msgr2=0x563ffe748200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.364+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3823699871 shutdown_connections 2024-03-24T00:45:26.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.364+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3823699871 wait complete. 2024-03-24T00:45:26.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.366+0000 7f6aadd9f740 1 -- start start 2024-03-24T00:45:26.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.367+0000 7f6aadd9f740 1 Processor -- start 2024-03-24T00:45:26.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 0x563ffe75cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe75d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 0x563ffe75e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b6550 con 0x563ffe74de80 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b7870 con 0x563ffe7539e0 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.369+0000 7f6aadd9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41b6bb0 con 0x563ffe74e870 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.370+0000 7f6aadd9f740 10 client.0 _ll_register_callbacks cb 0x7f6a9c098d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.370+0000 7f6aa9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 0x563ffe75e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.370+0000 7f6aa9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 0x563ffe75e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53614/0 (socket says 172.21.15.123:53614) 2024-03-24T00:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.370+0000 7f6aa9000640 1 -- 172.21.15.123:0/3061425801 learned_addr learned my addr 172.21.15.123:0/3061425801 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:26.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.370+0000 7f6aa3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe75d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.372+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 0x563ffe75cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.372+0000 7f6aa3400640 1 -- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 msgr2=0x563ffe75e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.372+0000 7f6aa3400640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 0x563ffe75e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.372+0000 7f6aa3400640 1 -- 172.21.15.123:0/3061425801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 msgr2=0x563ffe75cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:26.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.373+0000 7f6aa3400640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 0x563ffe75cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.374+0000 7f6aa3400640 1 -- 172.21.15.123:0/3061425801 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ffe75eac0 con 0x563ffe74e870 2024-03-24T00:45:26.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.375+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 0x563ffe75cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:26.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.376+0000 7f6aa3400640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe75d4c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6a900096c0 tx=0x7f6a9003a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=397235a70c78b57f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:26.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.376+0000 7f6a8e000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:45:26.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.377+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90052060 con 0x563ffe74e870 2024-03-24T00:45:26.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.378+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6a90032840 con 0x563ffe74e870 2024-03-24T00:45:26.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.378+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a94009780 con 0x563ffe74e870 2024-03-24T00:45:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.380+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a90039040 con 0x563ffe74e870 2024-03-24T00:45:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.380+0000 7f6aadd9f740 10 client.26488 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:26.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.380+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563ffe7d0a60 con 0x563ffe74e870 2024-03-24T00:45:26.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a94009780 con 0x563ffe74e870 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6a90003c20 con 0x563ffe74e870 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 10 client.26488 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 10 client.26488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 20 client.26488 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe7d1410 0x563ffe7d3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 4 client.26488 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.381+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563ffe6baae0 con 0x563ffe7d1410 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.382+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe7d1410 0x563ffe7d3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.383+0000 7f6aa3e00640 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe7d1410 0x563ffe7d3830 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb92432e90ab0488 server_cookie=57280b812f77f7ef in_seq=0 out_seq=0 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.384+0000 7f6a8e000640 10 client.26488 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.384+0000 7f6a8e000640 1 -- 172.21.15.123:0/3061425801 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x563ffe6baae0 con 0x563ffe7d1410 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.384+0000 7f6a8e000640 10 client.26488 handle_command_reply: tid=0 2024-03-24T00:45:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.384+0000 7f6aadd9f740 1 client.26488 shutdown 2024-03-24T00:45:26.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.385+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 msgr2=0x563ffe75d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.385+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe75d4c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6a900096c0 tx=0x7f6a9003a000 comp rx=0 tx=0).stop 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 shutdown_connections 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563ffe7539e0 0x563ffe75e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563ffe7d1410 0x563ffe7d3830 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563ffe74e870 0x563ffe75d4c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 --2- 172.21.15.123:0/3061425801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563ffe74de80 0x563ffe75cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 >> 172.21.15.123:0/3061425801 conn(0x563ffe716200 msgr2=0x563ffe748a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6a8e000640 0 client.26488 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:26.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 shutdown_connections 2024-03-24T00:45:26.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.386+0000 7f6aadd9f740 1 -- 172.21.15.123:0/3061425801 wait complete. 2024-03-24T00:45:26.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.387+0000 7f6aadd9f740 20 client.26488 trim_cache size 0 max 16384 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (39 inodes in the stack)", 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "762d32e1-29f5-4dbd-8d66-da389d05b26e": { 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "762d32e1-29f5-4dbd-8d66-da389d05b26e", 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:45:26.390 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:45:26.391 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:45:26.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.390+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 msgr2=0x7f6a880a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.390+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 0x7f6a880a4dc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6a90037040 tx=0x7f6a90000bc0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.390+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 msgr2=0x7f6aa41ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.390+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00cce0 tx=0x7f6a9c00a620 comp rx=0 tx=0).stop 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 shutdown_connections 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa4103280 0x7f6aa41ae260 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f6a880a2900 0x7f6a880a4dc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa41028c0 0x7f6aa4195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 --2- 172.21.15.123:0/4034973664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6aa41016d0 0x7f6aa4195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.391+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 >> 172.21.15.123:0/4034973664 conn(0x7f6aa40fcc40 msgr2=0x7f6aa4103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.392+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 shutdown_connections 2024-03-24T00:45:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:26.392+0000 7f6aab000640 1 -- 172.21.15.123:0/4034973664 wait complete. 2024-03-24T00:45:26.432 INFO:tasks.cephfs.filesystem:scrub status for tag:762d32e1-29f5-4dbd-8d66-da389d05b26e - {'path': '/', 'tag': '762d32e1-29f5-4dbd-8d66-da389d05b26e', 'options': 'recursive,force'} 2024-03-24T00:45:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:26 smithi177 ceph-mon[36034]: pgmap v611: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 36 MiB/s wr, 3.25k op/s 2024-03-24T00:45:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:26 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:45:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:26 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:26.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:26 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:45:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:26 smithi204 ceph-mon[36345]: pgmap v611: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 36 MiB/s wr, 3.25k op/s 2024-03-24T00:45:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:26 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:45:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:26 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:26 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:45:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:26 smithi123 ceph-mon[31487]: pgmap v611: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 36 MiB/s wr, 3.25k op/s 2024-03-24T00:45:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:26 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:45:26.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:26 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:26.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:26 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:45:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:28 smithi177 ceph-mon[36034]: pgmap v612: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 36 MiB/s wr, 3.36k op/s 2024-03-24T00:45:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:28 smithi204 ceph-mon[36345]: pgmap v612: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 36 MiB/s wr, 3.36k op/s 2024-03-24T00:45:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:28 smithi123 ceph-mon[31487]: pgmap v612: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 36 MiB/s wr, 3.36k op/s 2024-03-24T00:45:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:30 smithi177 ceph-mon[36034]: pgmap v613: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 35 MiB/s wr, 3.32k op/s 2024-03-24T00:45:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:30 smithi204 ceph-mon[36345]: pgmap v613: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 35 MiB/s wr, 3.32k op/s 2024-03-24T00:45:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:30 smithi123 ceph-mon[31487]: pgmap v613: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 35 MiB/s wr, 3.32k op/s 2024-03-24T00:45:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:32 smithi177 ceph-mon[36034]: pgmap v614: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 40 MiB/s wr, 3.83k op/s 2024-03-24T00:45:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:32 smithi204 ceph-mon[36345]: pgmap v614: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 40 MiB/s wr, 3.83k op/s 2024-03-24T00:45:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:32 smithi123 ceph-mon[31487]: pgmap v614: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 40 MiB/s wr, 3.83k op/s 2024-03-24T00:45:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 -- 172.21.15.123:0/2843957486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 msgr2=0x7feac0100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 --2- 172.21.15.123:0/2843957486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac0100200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feab0009900 tx=0x7feab002bcd0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 -- 172.21.15.123:0/2843957486 shutdown_connections 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 --2- 172.21.15.123:0/2843957486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac01010e0 0x7feac0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 --2- 172.21.15.123:0/2843957486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac0100740 0x7feac0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 --2- 172.21.15.123:0/2843957486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac0100200 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 -- 172.21.15.123:0/2843957486 >> 172.21.15.123:0/2843957486 conn(0x7feac00fb980 msgr2=0x7feac00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.838+0000 7feac5a00640 1 -- 172.21.15.123:0/2843957486 shutdown_connections 2024-03-24T00:45:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.839+0000 7feac5a00640 1 -- 172.21.15.123:0/2843957486 wait complete. 2024-03-24T00:45:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 Processor -- start 2024-03-24T00:45:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 -- start start 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac0100740 0x7feac0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac01010e0 0x7feac01a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac010f390 con 0x7feac0100740 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac010f210 con 0x7feac01010e0 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac010f510 con 0x7feac00ffde0 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feabfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feabfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54796/0 (socket says 172.21.15.123:54796) 2024-03-24T00:45:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feabfe00640 1 -- 172.21.15.123:0/2051523461 learned_addr learned my addr 172.21.15.123:0/2051523461 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feac5000640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac01010e0 0x7feac01a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.840+0000 7feabf400640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac0100740 0x7feac0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 -- 172.21.15.123:0/2051523461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac01010e0 msgr2=0x7feac01a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac01010e0 0x7feac01a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 -- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac0100740 msgr2=0x7feac0199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac0100740 0x7feac0199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 -- 172.21.15.123:0/2051523461 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac01a15f0 con 0x7feac00ffde0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabfe00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feab0000c00 tx=0x7feab002c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabcc00640 1 -- 172.21.15.123:0/2051523461 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0043020 con 0x7feac00ffde0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabcc00640 1 -- 172.21.15.123:0/2051523461 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab00054e0 con 0x7feac00ffde0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feabcc00640 1 -- 172.21.15.123:0/2051523461 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0003c20 con 0x7feac00ffde0 2024-03-24T00:45:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.841+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac01a1880 con 0x7feac00ffde0 2024-03-24T00:45:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.842+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac01a1d80 con 0x7feac00ffde0 2024-03-24T00:45:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.844+0000 7feabcc00640 1 -- 172.21.15.123:0/2051523461 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feab0046020 con 0x7feac00ffde0 2024-03-24T00:45:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.845+0000 7feac9dfc740 1 Processor -- start 2024-03-24T00:45:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 -- start start 2024-03-24T00:45:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabcc00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 0x7feaa40a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7c1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c6760 0x5585ad7c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad77b3b0 con 0x5585ad7c0c00 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad71d830 con 0x5585ad7c15f0 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac9dfc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad6c65d0 con 0x5585ad7c6760 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabcc00640 1 -- 172.21.15.123:0/2051523461 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feab00f0b30 con 0x7feac00ffde0 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c6760 0x5585ad7c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53648/0 (socket says 172.21.15.123:53648) 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabf400640 1 -- 172.21.15.123:0/3225902186 learned_addr learned my addr 172.21.15.123:0/3225902186 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feabfe00640 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.846+0000 7feac5000640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 0x7feaa40a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 -- 172.21.15.123:0/3225902186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c6760 msgr2=0x5585ad7c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c6760 0x5585ad7c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 -- 172.21.15.123:0/3225902186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 msgr2=0x5585ad7c1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7c1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 -- 172.21.15.123:0/3225902186 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ad6de2c0 con 0x5585ad7c15f0 2024-03-24T00:45:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feabf400640 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009c20 tx=0x7feaa80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a2c757db105ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feac5000640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 0x7feaa40a4d30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feab8005c40 tx=0x7feab8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.847+0000 7feab5600640 1 -- 172.21.15.123:0/3225902186 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8012450 con 0x5585ad7c15f0 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feab5600640 1 -- 172.21.15.123:0/3225902186 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaa8012ad0 con 0x5585ad7c15f0 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feab5600640 1 -- 172.21.15.123:0/3225902186 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa800fbf0 con 0x5585ad7c15f0 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 -- 172.21.15.123:0/3225902186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 msgr2=0x5585ad7c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009c20 tx=0x7feaa80117b0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 -- 172.21.15.123:0/3225902186 shutdown_connections 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c6760 0x5585ad7c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7c60f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3225902186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7c1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 -- 172.21.15.123:0/3225902186 >> 172.21.15.123:0/3225902186 conn(0x5585ad7b3960 msgr2=0x5585ad7b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.848+0000 7feac9dfc740 1 -- 172.21.15.123:0/3225902186 shutdown_connections 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.849+0000 7feac9dfc740 1 -- 172.21.15.123:0/3225902186 wait complete. 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.849+0000 7feac9dfc740 1 -- start start 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.849+0000 7feac9dfc740 1 Processor -- start 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad6d9a20 con 0x5585ad7c6760 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad7198d0 con 0x5585ad7c15f0 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad7c9710 con 0x5585ad7c0c00 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac9dfc740 10 client.0 _ll_register_callbacks cb 0x5585ad7161c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54816/0 (socket says 172.21.15.123:54816) 2024-03-24T00:45:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 -- 172.21.15.123:0/618716190 learned_addr learned my addr 172.21.15.123:0/618716190 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feac5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabf400640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 -- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 msgr2=0x5585ad7cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 -- 172.21.15.123:0/618716190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 msgr2=0x5585ad7d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.850+0000 7feabfe00640 1 -- 172.21.15.123:0/618716190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585ad432210 con 0x5585ad7c0c00 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feabfe00640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feab00ee5e0 tx=0x7feab00036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29bb049bc36b958d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab009aaf0 con 0x5585ad7c0c00 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feab00ee6f0 con 0x5585ad7c0c00 2024-03-24T00:45:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feaa8012ad0 con 0x5585ad7c0c00 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feab009a910 con 0x5585ad7c0c00 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feac9dfc740 10 client.26500 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5585ad4fe620 con 0x5585ad7c0c00 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaa8012ad0 con 0x5585ad7c0c00 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.851+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feab00d60d0 con 0x5585ad7c0c00 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 10 client.26500 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 10 client.26500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 20 client.26500 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad7d6c90 0x5585ad7d9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 4 client.26500 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585ad72ad70 con 0x5585ad7d6c90 2024-03-24T00:45:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.852+0000 7feabf400640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad7d6c90 0x5585ad7d9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.853+0000 7feaae000640 10 client.26500 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.853+0000 7feabf400640 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad7d6c90 0x5585ad7d9100 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5425503feadb4f69 server_cookie=1589b60bb1f71b37 in_seq=0 out_seq=0 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feaae000640 1 -- 172.21.15.123:0/618716190 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5585ad72ad70 con 0x5585ad7d6c90 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feaae000640 10 client.26500 handle_command_reply: tid=0 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 client.26500 shutdown 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 msgr2=0x5585ad7cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feab00ee5e0 tx=0x7feab00036f0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 shutdown_connections 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c15f0 0x5585ad7cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad7d6c90 0x5585ad7d9100 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 --2- 172.21.15.123:0/618716190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c0c00 0x5585ad7cae40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 >> 172.21.15.123:0/618716190 conn(0x5585ad7b3960 msgr2=0x5585ad7b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feaae000640 0 client.26500 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 shutdown_connections 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 1 -- 172.21.15.123:0/618716190 wait complete. 2024-03-24T00:45:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.854+0000 7feac9dfc740 20 client.26500 trim_cache size 0 max 16384 2024-03-24T00:45:33.871 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 Processor -- start 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 -- start start 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a4690 con 0x5585ad7c0c00 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a6c30 con 0x5585ad7c6760 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.870+0000 7feac9dfc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a4c90 con 0x5585ad7c15f0 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53668/0 (socket says 172.21.15.123:53668) 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 -- 172.21.15.123:0/3298921816 learned_addr learned my addr 172.21.15.123:0/3298921816 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feabf400640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feabfe00640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 -- 172.21.15.123:0/3298921816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 msgr2=0x5585ad7d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 -- 172.21.15.123:0/3298921816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 msgr2=0x5585ad7b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feac5000640 1 -- 172.21.15.123:0/3298921816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ad7b1f80 con 0x5585ad7c6760 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.871+0000 7feabf400640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.872+0000 7feac5000640 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7feab8011ab0 tx=0x7feab8013070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a457fa908901ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.872+0000 7feab4c00640 1 -- 172.21.15.123:0/3298921816 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab800fcc0 con 0x5585ad7c6760 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.872+0000 7feab4c00640 1 -- 172.21.15.123:0/3298921816 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab800fe60 con 0x5585ad7c6760 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.872+0000 7feab4c00640 1 -- 172.21.15.123:0/3298921816 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8005720 con 0x5585ad7c6760 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- 172.21.15.123:0/3298921816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 msgr2=0x5585ad7b7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7feab8011ab0 tx=0x7feab8013070 comp rx=0 tx=0).stop 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- 172.21.15.123:0/3298921816 shutdown_connections 2024-03-24T00:45:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c6760 0x5585ad7b7100 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 --2- 172.21.15.123:0/3298921816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c0c00 0x5585ad7b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- 172.21.15.123:0/3298921816 >> 172.21.15.123:0/3298921816 conn(0x5585ad7b3960 msgr2=0x5585ad7b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- 172.21.15.123:0/3298921816 shutdown_connections 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- 172.21.15.123:0/3298921816 wait complete. 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 -- start start 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.873+0000 7feac9dfc740 1 Processor -- start 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a52b0 con 0x5585ad7c6760 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a65d0 con 0x5585ad7c0c00 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01a5910 con 0x5585ad7c15f0 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac9dfc740 10 client.0 _ll_register_callbacks cb 0x7feab009c0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54838/0 (socket says 172.21.15.123:54838) 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:53676/0 (socket says 172.21.15.123:53676) 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 -- 172.21.15.123:0/731914596 learned_addr learned my addr 172.21.15.123:0/731914596 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac5000640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 -- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 msgr2=0x5585ad7cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 -- 172.21.15.123:0/731914596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 msgr2=0x5585ad7d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabf400640 1 -- 172.21.15.123:0/731914596 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585ad7d1770 con 0x5585ad7c15f0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feabfe00640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.874+0000 7feac5000640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feabf400640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5585ad7c2060 tx=0x7feaa80052e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bffaa366c023aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feaae000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa80037f0 con 0x5585ad7c15f0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feab800fe60 con 0x5585ad7c15f0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feaa8003990 con 0x5585ad7c15f0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feaa8009e30 con 0x5585ad7c15f0 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feac9dfc740 10 client.26506 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.875+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5585ad832840 con 0x5585ad7c15f0 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab800fe60 con 0x5585ad7c15f0 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feaa800fb00 con 0x5585ad7c15f0 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 10 client.26506 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 20 client.26506 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad832f10 0x5585ad835330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 4 client.26506 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5585ad7250c0 con 0x5585ad832f10 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.876+0000 7feabfe00640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad832f10 0x5585ad835330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.877+0000 7feabfe00640 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad832f10 0x5585ad835330 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11b160016166d8bc server_cookie=eb41671278aa064c in_seq=0 out_seq=0 2024-03-24T00:45:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:45:33.877+0000 7feaae000640 10 client.26506 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:34.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:34 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:34.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:34 smithi177 ceph-mon[36034]: pgmap v615: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 36 MiB/s wr, 3.50k op/s 2024-03-24T00:45:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:34 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:34 smithi204 ceph-mon[36345]: pgmap v615: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 36 MiB/s wr, 3.50k op/s 2024-03-24T00:45:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:34 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:45:34.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:34 smithi123 ceph-mon[31487]: pgmap v615: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 36 MiB/s wr, 3.50k op/s 2024-03-24T00:45:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:36 smithi177 ceph-mon[36034]: pgmap v616: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 36 MiB/s wr, 3.55k op/s 2024-03-24T00:45:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:36 smithi204 ceph-mon[36345]: pgmap v616: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 36 MiB/s wr, 3.55k op/s 2024-03-24T00:45:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:36 smithi123 ceph-mon[31487]: pgmap v616: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 36 MiB/s wr, 3.55k op/s 2024-03-24T00:45:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:37 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:45:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:37 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:45:37.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:37 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:45:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:38 smithi204 ceph-mon[36345]: pgmap v617: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 35 MiB/s wr, 3.42k op/s 2024-03-24T00:45:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:38 smithi123 ceph-mon[31487]: pgmap v617: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 35 MiB/s wr, 3.42k op/s 2024-03-24T00:45:38.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:38 smithi177 ceph-mon[36034]: pgmap v617: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 35 MiB/s wr, 3.42k op/s 2024-03-24T00:45:39.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:39 smithi204 ceph-mon[36345]: pgmap v618: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 32 MiB/s wr, 3.10k op/s 2024-03-24T00:45:39.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:39 smithi123 ceph-mon[31487]: pgmap v618: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 32 MiB/s wr, 3.10k op/s 2024-03-24T00:45:39.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:39 smithi177 ceph-mon[36034]: pgmap v618: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 32 MiB/s wr, 3.10k op/s 2024-03-24T00:45:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:42 smithi177 ceph-mon[36034]: pgmap v619: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 36 MiB/s wr, 3.61k op/s 2024-03-24T00:45:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:42 smithi204 ceph-mon[36345]: pgmap v619: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 36 MiB/s wr, 3.61k op/s 2024-03-24T00:45:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:42 smithi123 ceph-mon[31487]: pgmap v619: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 36 MiB/s wr, 3.61k op/s 2024-03-24T00:45:44.660 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:44 smithi123 ceph-mon[31487]: pgmap v620: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 31 MiB/s wr, 3.20k op/s 2024-03-24T00:45:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:44 smithi177 ceph-mon[36034]: pgmap v620: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 31 MiB/s wr, 3.20k op/s 2024-03-24T00:45:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:44 smithi204 ceph-mon[36345]: pgmap v620: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 31 MiB/s wr, 3.20k op/s 2024-03-24T00:45:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:46 smithi177 ceph-mon[36034]: pgmap v621: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 31 MiB/s wr, 3.18k op/s 2024-03-24T00:45:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:46 smithi204 ceph-mon[36345]: pgmap v621: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 31 MiB/s wr, 3.18k op/s 2024-03-24T00:45:46.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:46 smithi123 ceph-mon[31487]: pgmap v621: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 31 MiB/s wr, 3.18k op/s 2024-03-24T00:45:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:48 smithi177 ceph-mon[36034]: pgmap v622: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 32 MiB/s wr, 3.16k op/s 2024-03-24T00:45:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:48 smithi204 ceph-mon[36345]: pgmap v622: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 32 MiB/s wr, 3.16k op/s 2024-03-24T00:45:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:48 smithi123 ceph-mon[31487]: pgmap v622: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 32 MiB/s wr, 3.16k op/s 2024-03-24T00:45:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:50 smithi177 ceph-mon[36034]: pgmap v623: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 34 MiB/s wr, 3.39k op/s 2024-03-24T00:45:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:50 smithi204 ceph-mon[36345]: pgmap v623: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 34 MiB/s wr, 3.39k op/s 2024-03-24T00:45:50.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:50 smithi123 ceph-mon[31487]: pgmap v623: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 34 MiB/s wr, 3.39k op/s 2024-03-24T00:45:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:52 smithi177 ceph-mon[36034]: pgmap v624: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 50 MiB/s wr, 4.88k op/s 2024-03-24T00:45:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:52 smithi204 ceph-mon[36345]: pgmap v624: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 50 MiB/s wr, 4.88k op/s 2024-03-24T00:45:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:52 smithi123 ceph-mon[31487]: pgmap v624: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 50 MiB/s wr, 4.88k op/s 2024-03-24T00:45:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:54 smithi177 ceph-mon[36034]: pgmap v625: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 45 MiB/s wr, 4.40k op/s 2024-03-24T00:45:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:54 smithi204 ceph-mon[36345]: pgmap v625: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 45 MiB/s wr, 4.40k op/s 2024-03-24T00:45:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:54 smithi123 ceph-mon[31487]: pgmap v625: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 45 MiB/s wr, 4.40k op/s 2024-03-24T00:45:56.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:45:56.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:45:56.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 -- 172.21.15.123:0/1303685989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 msgr2=0x7f058c1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:56.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 --2- 172.21.15.123:0/1303685989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c1019c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0580009900 tx=0x7f058002bc30 comp rx=0 tx=0).stop 2024-03-24T00:45:56.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 -- 172.21.15.123:0/1303685989 shutdown_connections 2024-03-24T00:45:56.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 --2- 172.21.15.123:0/1303685989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 0x7f058c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 --2- 172.21.15.123:0/1303685989 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 --2- 172.21.15.123:0/1303685989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c1019c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 -- 172.21.15.123:0/1303685989 >> 172.21.15.123:0/1303685989 conn(0x7f058c0fcb10 msgr2=0x7f058c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:56.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 -- 172.21.15.123:0/1303685989 shutdown_connections 2024-03-24T00:45:56.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.666+0000 7f0592400640 1 -- 172.21.15.123:0/1303685989 wait complete. 2024-03-24T00:45:56.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.667+0000 7f0592400640 1 Processor -- start 2024-03-24T00:45:56.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.667+0000 7f0592400640 1 -- start start 2024-03-24T00:45:56.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.667+0000 7f0592400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.667+0000 7f0592400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0592400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 0x7f058c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0592400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c10d260 con 0x7f058c103150 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0592400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c10d0e0 con 0x7f058c1015a0 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0592400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c10d3e0 con 0x7f058c102790 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54884/0 (socket says 172.21.15.123:54884) 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:55466/0 (socket says 172.21.15.123:55466) 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0591000640 1 -- 172.21.15.123:0/2607431526 learned_addr learned my addr 172.21.15.123:0/2607431526 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0591a00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 0x7f058c1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 -- 172.21.15.123:0/2607431526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 msgr2=0x7f058c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 -- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 msgr2=0x7f058c1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:56.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 0x7f058c1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f058be00640 1 -- 172.21.15.123:0/2607431526 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c1a1770 con 0x7f058c102790 2024-03-24T00:45:56.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.668+0000 7f0591000640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f058be00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f057800e9f0 tx=0x7f057800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f0589600640 1 -- 172.21.15.123:0/2607431526 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057800ccc0 con 0x7f058c102790 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c1a1a60 con 0x7f058c102790 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c1a1f60 con 0x7f058c102790 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f0589600640 1 -- 172.21.15.123:0/2607431526 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0578005020 con 0x7f058c102790 2024-03-24T00:45:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.669+0000 7f0589600640 1 -- 172.21.15.123:0/2607431526 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780049e0 con 0x7f058c102790 2024-03-24T00:45:56.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.674+0000 7f0596789740 1 Processor -- start 2024-03-24T00:45:56.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.674+0000 7f0589600640 1 -- 172.21.15.123:0/2607431526 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0578004cc0 con 0x7f058c102790 2024-03-24T00:45:56.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.674+0000 7f0596789740 1 -- start start 2024-03-24T00:45:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.675+0000 7f0596789740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.675+0000 7f0596789740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a76cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.675+0000 7f0596789740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a76d3c0 0x56384a76f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.675+0000 7f0596789740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a6c4250 con 0x56384a768250 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0596789740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a66ca30 con 0x56384a76d3c0 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0596789740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a6c02f0 con 0x56384a767860 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0591a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0591a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:55472/0 (socket says 172.21.15.123:55472) 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0591a00640 1 -- 172.21.15.123:0/447955349 learned_addr learned my addr 172.21.15.123:0/447955349 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:56.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0591000640 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a76cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.676+0000 7f0589600640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0589600640 1 -- 172.21.15.123:0/2607431526 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f05780ce9e0 con 0x7f058c102790 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 -- 172.21.15.123:0/447955349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a76d3c0 msgr2=0x56384a76f850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a76d3c0 0x56384a76f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 -- 172.21.15.123:0/447955349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 msgr2=0x56384a76cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:56.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a76cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 -- 172.21.15.123:0/447955349 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56384a657690 con 0x56384a767860 2024-03-24T00:45:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.677+0000 7f0591a00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.678+0000 7f0591a00640 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f058400f480 tx=0x7f058400f940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca2e5fbe2c93d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:56.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.678+0000 7f0591a00640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 0x7f05740a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f058400fe40 tx=0x7f0584014000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.679+0000 7f057e000640 1 -- 172.21.15.123:0/447955349 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584012d20 con 0x56384a767860 2024-03-24T00:45:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.679+0000 7f057e000640 1 -- 172.21.15.123:0/447955349 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f058400dd20 con 0x56384a767860 2024-03-24T00:45:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.679+0000 7f057e000640 1 -- 172.21.15.123:0/447955349 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840103d0 con 0x56384a767860 2024-03-24T00:45:56.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.679+0000 7f0596789740 1 -- 172.21.15.123:0/447955349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 msgr2=0x56384a767c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.679+0000 7f0596789740 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f058400f480 tx=0x7f058400f940 comp rx=0 tx=0).stop 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 -- 172.21.15.123:0/447955349 shutdown_connections 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a76d3c0 0x56384a76f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a76cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 --2- 172.21.15.123:0/447955349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a767c80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 -- 172.21.15.123:0/447955349 >> 172.21.15.123:0/447955349 conn(0x56384a727200 msgr2=0x56384a7631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 -- 172.21.15.123:0/447955349 shutdown_connections 2024-03-24T00:45:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 -- 172.21.15.123:0/447955349 wait complete. 2024-03-24T00:45:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.680+0000 7f0596789740 1 -- start start 2024-03-24T00:45:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 Processor -- start 2024-03-24T00:45:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a768250 0x56384a771e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a778e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a770370 con 0x56384a76d3c0 2024-03-24T00:45:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a67fe80 con 0x56384a767860 2024-03-24T00:45:56.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384a7704f0 con 0x56384a768250 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.681+0000 7f0596789740 10 client.0 _ll_register_callbacks cb 0x56384a692130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:54902/0 (socket says 172.21.15.123:54902) 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 -- 172.21.15.123:0/117098499 learned_addr learned my addr 172.21.15.123:0/117098499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591a00640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a778e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 -- 172.21.15.123:0/117098499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a768250 msgr2=0x56384a771e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a768250 0x56384a771e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 -- 172.21.15.123:0/117098499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 msgr2=0x56384a778e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a778e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.682+0000 7f0591000640 1 -- 172.21.15.123:0/117098499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56384a4a27d0 con 0x56384a767860 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0591000640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f058002c170 tx=0x7f05800048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9176cb47d65efb91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:56.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0572a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580005d50 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0580004ed0 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f058400dd20 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f058002cd30 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.683+0000 7f0596789740 10 client.26504 fetch_fsmap learned FSMap version 19 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56384a3d83f0 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f058400dd20 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0580003b80 con 0x56384a767860 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 10 client.26504 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 10 client.26504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 20 client.26504 populate_metadata read hostname 'smithi123' 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a77d770 0x56384a77fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 4 client.26504 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:56.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56384a6d1790 con 0x56384a77d770 2024-03-24T00:45:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.684+0000 7f058be00640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a77d770 0x56384a77fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.685+0000 7f058be00640 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a77d770 0x56384a77fbd0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3140a8b05303ca57 server_cookie=fdf7e705c4e4b061 in_seq=0 out_seq=0 2024-03-24T00:45:56.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:45:56.685+0000 7f0572a00640 10 client.26504 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:45:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:56 smithi177 ceph-mon[36034]: pgmap v626: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 53 MiB/s wr, 5.28k op/s 2024-03-24T00:45:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:56 smithi204 ceph-mon[36345]: pgmap v626: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 53 MiB/s wr, 5.28k op/s 2024-03-24T00:45:56.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:56 smithi123 ceph-mon[31487]: pgmap v626: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 53 MiB/s wr, 5.28k op/s 2024-03-24T00:45:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:45:58 smithi177 ceph-mon[36034]: pgmap v627: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 54 MiB/s wr, 5.36k op/s 2024-03-24T00:45:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:45:58 smithi204 ceph-mon[36345]: pgmap v627: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 54 MiB/s wr, 5.36k op/s 2024-03-24T00:45:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:45:58 smithi123 ceph-mon[31487]: pgmap v627: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 54 MiB/s wr, 5.36k op/s 2024-03-24T00:46:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:00 smithi204 ceph-mon[36345]: pgmap v628: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 56 MiB/s wr, 5.74k op/s 2024-03-24T00:46:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:00 smithi123 ceph-mon[31487]: pgmap v628: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 56 MiB/s wr, 5.74k op/s 2024-03-24T00:46:00.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:00 smithi177 ceph-mon[36034]: pgmap v628: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 56 MiB/s wr, 5.74k op/s 2024-03-24T00:46:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:02 smithi204 ceph-mon[36345]: pgmap v629: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 57 MiB/s wr, 5.99k op/s 2024-03-24T00:46:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:02 smithi123 ceph-mon[31487]: pgmap v629: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 57 MiB/s wr, 5.99k op/s 2024-03-24T00:46:02.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:02 smithi177 ceph-mon[36034]: pgmap v629: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 57 MiB/s wr, 5.99k op/s 2024-03-24T00:46:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:04 smithi204 ceph-mon[36345]: pgmap v630: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 41 MiB/s wr, 4.52k op/s 2024-03-24T00:46:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:04 smithi123 ceph-mon[31487]: pgmap v630: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 41 MiB/s wr, 4.52k op/s 2024-03-24T00:46:04.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:04 smithi177 ceph-mon[36034]: pgmap v630: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 41 MiB/s wr, 4.52k op/s 2024-03-24T00:46:05.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:05 smithi204 ceph-mon[36345]: pgmap v631: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 47 MiB/s wr, 5.17k op/s 2024-03-24T00:46:05.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:05 smithi123 ceph-mon[31487]: pgmap v631: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 47 MiB/s wr, 5.17k op/s 2024-03-24T00:46:05.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:05 smithi177 ceph-mon[36034]: pgmap v631: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 47 MiB/s wr, 5.17k op/s 2024-03-24T00:46:08.664 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:08 smithi177 ceph-mon[36034]: pgmap v632: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 39 MiB/s wr, 4.46k op/s 2024-03-24T00:46:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:08 smithi204 ceph-mon[36345]: pgmap v632: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 39 MiB/s wr, 4.46k op/s 2024-03-24T00:46:08.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:08 smithi123 ceph-mon[31487]: pgmap v632: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 39 MiB/s wr, 4.46k op/s 2024-03-24T00:46:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:10 smithi177 ceph-mon[36034]: pgmap v633: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 4.54k op/s 2024-03-24T00:46:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:10 smithi204 ceph-mon[36345]: pgmap v633: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 4.54k op/s 2024-03-24T00:46:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:10 smithi123 ceph-mon[31487]: pgmap v633: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 4.54k op/s 2024-03-24T00:46:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:12 smithi177 ceph-mon[36034]: pgmap v634: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.24k op/s 2024-03-24T00:46:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:12 smithi204 ceph-mon[36345]: pgmap v634: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.24k op/s 2024-03-24T00:46:12.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:12 smithi123 ceph-mon[31487]: pgmap v634: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.24k op/s 2024-03-24T00:46:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:14 smithi177 ceph-mon[36034]: pgmap v635: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 3.65k op/s 2024-03-24T00:46:14.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:14 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:14 smithi204 ceph-mon[36345]: pgmap v635: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 3.65k op/s 2024-03-24T00:46:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:14 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:14 smithi123 ceph-mon[31487]: pgmap v635: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 3.65k op/s 2024-03-24T00:46:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:14 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:15.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:15 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:15.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:15 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:15.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:15 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:46:15.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:15 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:15.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:15 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:15.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:15 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:46:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:15.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:46:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:16 smithi177 ceph-mon[36034]: pgmap v636: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 4.25k op/s 2024-03-24T00:46:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:16 smithi204 ceph-mon[36345]: pgmap v636: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 4.25k op/s 2024-03-24T00:46:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:16 smithi123 ceph-mon[31487]: pgmap v636: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 4.25k op/s 2024-03-24T00:46:18.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:18 smithi177 ceph-mon[36034]: pgmap v637: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 31 MiB/s wr, 3.72k op/s 2024-03-24T00:46:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:18 smithi204 ceph-mon[36345]: pgmap v637: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 31 MiB/s wr, 3.72k op/s 2024-03-24T00:46:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:18 smithi123 ceph-mon[31487]: pgmap v637: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 31 MiB/s wr, 3.72k op/s 2024-03-24T00:46:20.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:20 smithi177 ceph-mon[36034]: pgmap v638: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 32 MiB/s wr, 3.74k op/s 2024-03-24T00:46:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:20 smithi204 ceph-mon[36345]: pgmap v638: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 32 MiB/s wr, 3.74k op/s 2024-03-24T00:46:20.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:20 smithi123 ceph-mon[31487]: pgmap v638: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 32 MiB/s wr, 3.74k op/s 2024-03-24T00:46:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:22 smithi177 ceph-mon[36034]: pgmap v639: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 4.05k op/s 2024-03-24T00:46:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:22 smithi204 ceph-mon[36345]: pgmap v639: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 4.05k op/s 2024-03-24T00:46:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:22 smithi123 ceph-mon[31487]: pgmap v639: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 4.05k op/s 2024-03-24T00:46:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:24 smithi177 ceph-mon[36034]: pgmap v640: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 33 MiB/s wr, 3.98k op/s 2024-03-24T00:46:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:24 smithi204 ceph-mon[36345]: pgmap v640: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 33 MiB/s wr, 3.98k op/s 2024-03-24T00:46:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:24 smithi123 ceph-mon[31487]: pgmap v640: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 33 MiB/s wr, 3.98k op/s 2024-03-24T00:46:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:26 smithi177 ceph-mon[36034]: pgmap v641: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 40 MiB/s wr, 4.86k op/s 2024-03-24T00:46:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:26 smithi204 ceph-mon[36345]: pgmap v641: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 40 MiB/s wr, 4.86k op/s 2024-03-24T00:46:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:26 smithi123 ceph-mon[31487]: pgmap v641: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 40 MiB/s wr, 4.86k op/s 2024-03-24T00:46:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:28 smithi204 ceph-mon[36345]: pgmap v642: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 4.38k op/s 2024-03-24T00:46:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:28 smithi123 ceph-mon[31487]: pgmap v642: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 4.38k op/s 2024-03-24T00:46:28.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:28 smithi177 ceph-mon[36034]: pgmap v642: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 4.38k op/s 2024-03-24T00:46:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:30 smithi204 ceph-mon[36345]: pgmap v643: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 34 MiB/s wr, 4.21k op/s 2024-03-24T00:46:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:30 smithi123 ceph-mon[31487]: pgmap v643: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 34 MiB/s wr, 4.21k op/s 2024-03-24T00:46:30.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:30 smithi177 ceph-mon[36034]: pgmap v643: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 34 MiB/s wr, 4.21k op/s 2024-03-24T00:46:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:32 smithi204 ceph-mon[36345]: pgmap v644: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 36 MiB/s wr, 4.52k op/s 2024-03-24T00:46:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:32 smithi123 ceph-mon[31487]: pgmap v644: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 36 MiB/s wr, 4.52k op/s 2024-03-24T00:46:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:32 smithi177 ceph-mon[36034]: pgmap v644: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 36 MiB/s wr, 4.52k op/s 2024-03-24T00:46:33.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:33 smithi204 ceph-mon[36345]: pgmap v645: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 31 MiB/s wr, 3.99k op/s 2024-03-24T00:46:33.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:33 smithi123 ceph-mon[31487]: pgmap v645: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 31 MiB/s wr, 3.99k op/s 2024-03-24T00:46:33.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:33 smithi177 ceph-mon[36034]: pgmap v645: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 31 MiB/s wr, 3.99k op/s 2024-03-24T00:46:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:36 smithi177 ceph-mon[36034]: pgmap v646: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 4.23k op/s 2024-03-24T00:46:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:36 smithi204 ceph-mon[36345]: pgmap v646: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 4.23k op/s 2024-03-24T00:46:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:36 smithi123 ceph-mon[31487]: pgmap v646: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 4.23k op/s 2024-03-24T00:46:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:38 smithi177 ceph-mon[36034]: pgmap v647: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 3.85k op/s 2024-03-24T00:46:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:38 smithi204 ceph-mon[36345]: pgmap v647: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 3.85k op/s 2024-03-24T00:46:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:38 smithi123 ceph-mon[31487]: pgmap v647: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 3.85k op/s 2024-03-24T00:46:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:40 smithi177 ceph-mon[36034]: pgmap v648: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.05k op/s 2024-03-24T00:46:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:40 smithi204 ceph-mon[36345]: pgmap v648: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.05k op/s 2024-03-24T00:46:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:40 smithi123 ceph-mon[31487]: pgmap v648: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.05k op/s 2024-03-24T00:46:42.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:42 smithi177 ceph-mon[36034]: pgmap v649: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 34 MiB/s wr, 4.61k op/s 2024-03-24T00:46:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:42 smithi204 ceph-mon[36345]: pgmap v649: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 34 MiB/s wr, 4.61k op/s 2024-03-24T00:46:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:42 smithi123 ceph-mon[31487]: pgmap v649: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 34 MiB/s wr, 4.61k op/s 2024-03-24T00:46:44.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:44 smithi177 ceph-mon[36034]: pgmap v650: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.16k op/s 2024-03-24T00:46:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:44 smithi204 ceph-mon[36345]: pgmap v650: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.16k op/s 2024-03-24T00:46:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:44 smithi123 ceph-mon[31487]: pgmap v650: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 31 MiB/s wr, 4.16k op/s 2024-03-24T00:46:46.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:46 smithi177 ceph-mon[36034]: pgmap v651: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 35 MiB/s wr, 4.81k op/s 2024-03-24T00:46:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:46 smithi204 ceph-mon[36345]: pgmap v651: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 35 MiB/s wr, 4.81k op/s 2024-03-24T00:46:46.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:46 smithi123 ceph-mon[31487]: pgmap v651: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 35 MiB/s wr, 4.81k op/s 2024-03-24T00:46:48.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:48 smithi177 ceph-mon[36034]: pgmap v652: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 33 MiB/s wr, 4.50k op/s 2024-03-24T00:46:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:48 smithi204 ceph-mon[36345]: pgmap v652: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 33 MiB/s wr, 4.50k op/s 2024-03-24T00:46:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:48 smithi123 ceph-mon[31487]: pgmap v652: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 33 MiB/s wr, 4.50k op/s 2024-03-24T00:46:50.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:50 smithi177 ceph-mon[36034]: pgmap v653: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 31 MiB/s wr, 4.28k op/s 2024-03-24T00:46:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:50 smithi204 ceph-mon[36345]: pgmap v653: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 31 MiB/s wr, 4.28k op/s 2024-03-24T00:46:50.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:50 smithi123 ceph-mon[31487]: pgmap v653: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 31 MiB/s wr, 4.28k op/s 2024-03-24T00:46:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:52 smithi204 ceph-mon[36345]: pgmap v654: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 4.84k op/s 2024-03-24T00:46:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:52 smithi123 ceph-mon[31487]: pgmap v654: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 4.84k op/s 2024-03-24T00:46:52.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:52 smithi177 ceph-mon[36034]: pgmap v654: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 4.84k op/s 2024-03-24T00:46:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:54 smithi204 ceph-mon[36345]: pgmap v655: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 31 MiB/s wr, 4.22k op/s 2024-03-24T00:46:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:54 smithi123 ceph-mon[31487]: pgmap v655: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 31 MiB/s wr, 4.22k op/s 2024-03-24T00:46:54.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:54 smithi177 ceph-mon[36034]: pgmap v655: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 31 MiB/s wr, 4.22k op/s 2024-03-24T00:46:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:56 smithi204 ceph-mon[36345]: pgmap v656: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.07k op/s 2024-03-24T00:46:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:56 smithi123 ceph-mon[31487]: pgmap v656: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.07k op/s 2024-03-24T00:46:56.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:56 smithi177 ceph-mon[36034]: pgmap v656: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.07k op/s 2024-03-24T00:46:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:46:57 smithi204 ceph-mon[36345]: pgmap v657: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 30 MiB/s wr, 4.34k op/s 2024-03-24T00:46:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:46:57 smithi123 ceph-mon[31487]: pgmap v657: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 30 MiB/s wr, 4.34k op/s 2024-03-24T00:46:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:46:57 smithi177 ceph-mon[36034]: pgmap v657: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 30 MiB/s wr, 4.34k op/s 2024-03-24T00:47:00.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:00 smithi177 ceph-mon[36034]: pgmap v658: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 29 MiB/s wr, 4.21k op/s 2024-03-24T00:47:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:00 smithi204 ceph-mon[36345]: pgmap v658: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 29 MiB/s wr, 4.21k op/s 2024-03-24T00:47:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:00 smithi123 ceph-mon[31487]: pgmap v658: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 29 MiB/s wr, 4.21k op/s 2024-03-24T00:47:02.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:02 smithi177 ceph-mon[36034]: pgmap v659: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 35 MiB/s wr, 5.12k op/s 2024-03-24T00:47:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:02 smithi204 ceph-mon[36345]: pgmap v659: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 35 MiB/s wr, 5.12k op/s 2024-03-24T00:47:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:02 smithi123 ceph-mon[31487]: pgmap v659: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 35 MiB/s wr, 5.12k op/s 2024-03-24T00:47:04.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:04 smithi177 ceph-mon[36034]: pgmap v660: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.27k op/s 2024-03-24T00:47:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:04 smithi204 ceph-mon[36345]: pgmap v660: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.27k op/s 2024-03-24T00:47:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:04 smithi123 ceph-mon[31487]: pgmap v660: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.27k op/s 2024-03-24T00:47:06.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:06 smithi177 ceph-mon[36034]: pgmap v661: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 33 MiB/s wr, 4.94k op/s 2024-03-24T00:47:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:06 smithi204 ceph-mon[36345]: pgmap v661: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 33 MiB/s wr, 4.94k op/s 2024-03-24T00:47:06.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:06 smithi123 ceph-mon[31487]: pgmap v661: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 33 MiB/s wr, 4.94k op/s 2024-03-24T00:47:08.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:08 smithi177 ceph-mon[36034]: pgmap v662: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.14k op/s 2024-03-24T00:47:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:08 smithi204 ceph-mon[36345]: pgmap v662: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.14k op/s 2024-03-24T00:47:08.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:08 smithi123 ceph-mon[31487]: pgmap v662: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 29 MiB/s wr, 4.14k op/s 2024-03-24T00:47:10.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:10 smithi177 ceph-mon[36034]: pgmap v663: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 28 MiB/s wr, 4.00k op/s 2024-03-24T00:47:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:10 smithi204 ceph-mon[36345]: pgmap v663: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 28 MiB/s wr, 4.00k op/s 2024-03-24T00:47:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:10 smithi123 ceph-mon[31487]: pgmap v663: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 28 MiB/s wr, 4.00k op/s 2024-03-24T00:47:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:12 smithi177 ceph-mon[36034]: pgmap v664: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 32 MiB/s wr, 4.71k op/s 2024-03-24T00:47:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:12 smithi204 ceph-mon[36345]: pgmap v664: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 32 MiB/s wr, 4.71k op/s 2024-03-24T00:47:12.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:12 smithi123 ceph-mon[31487]: pgmap v664: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 32 MiB/s wr, 4.71k op/s 2024-03-24T00:47:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:14 smithi177 ceph-mon[36034]: pgmap v665: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 27 MiB/s wr, 3.90k op/s 2024-03-24T00:47:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:14 smithi204 ceph-mon[36345]: pgmap v665: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 27 MiB/s wr, 3.90k op/s 2024-03-24T00:47:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:14 smithi123 ceph-mon[31487]: pgmap v665: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 27 MiB/s wr, 3.90k op/s 2024-03-24T00:47:15.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:15 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:15.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:15 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:15.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:15 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:16 smithi204 ceph-mon[36345]: pgmap v666: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 32 MiB/s wr, 4.73k op/s 2024-03-24T00:47:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:16 smithi123 ceph-mon[31487]: pgmap v666: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 32 MiB/s wr, 4.73k op/s 2024-03-24T00:47:16.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:16 smithi177 ceph-mon[36034]: pgmap v666: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 32 MiB/s wr, 4.73k op/s 2024-03-24T00:47:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:17.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:47:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:47:17.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:17.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:47:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:18 smithi204 ceph-mon[36345]: pgmap v667: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 30 MiB/s wr, 4.45k op/s 2024-03-24T00:47:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:18 smithi123 ceph-mon[31487]: pgmap v667: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 30 MiB/s wr, 4.45k op/s 2024-03-24T00:47:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:18 smithi177 ceph-mon[36034]: pgmap v667: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 30 MiB/s wr, 4.45k op/s 2024-03-24T00:47:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:20 smithi204 ceph-mon[36345]: pgmap v668: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 31 MiB/s wr, 4.62k op/s 2024-03-24T00:47:20.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:20 smithi123 ceph-mon[31487]: pgmap v668: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 31 MiB/s wr, 4.62k op/s 2024-03-24T00:47:20.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:20 smithi177 ceph-mon[36034]: pgmap v668: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 31 MiB/s wr, 4.62k op/s 2024-03-24T00:47:21.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:21 smithi204 ceph-mon[36345]: pgmap v669: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 34 MiB/s wr, 5.19k op/s 2024-03-24T00:47:21.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:21 smithi123 ceph-mon[31487]: pgmap v669: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 34 MiB/s wr, 5.19k op/s 2024-03-24T00:47:21.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:21 smithi177 ceph-mon[36034]: pgmap v669: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 34 MiB/s wr, 5.19k op/s 2024-03-24T00:47:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:24 smithi177 ceph-mon[36034]: pgmap v670: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 29 MiB/s wr, 4.48k op/s 2024-03-24T00:47:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:24 smithi204 ceph-mon[36345]: pgmap v670: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 29 MiB/s wr, 4.48k op/s 2024-03-24T00:47:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:24 smithi123 ceph-mon[31487]: pgmap v670: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 29 MiB/s wr, 4.48k op/s 2024-03-24T00:47:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:26 smithi177 ceph-mon[36034]: pgmap v671: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 33 MiB/s wr, 5.21k op/s 2024-03-24T00:47:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:26 smithi204 ceph-mon[36345]: pgmap v671: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 33 MiB/s wr, 5.21k op/s 2024-03-24T00:47:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:26 smithi123 ceph-mon[31487]: pgmap v671: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 33 MiB/s wr, 5.21k op/s 2024-03-24T00:47:28.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:28 smithi177 ceph-mon[36034]: pgmap v672: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 29 MiB/s wr, 4.59k op/s 2024-03-24T00:47:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:28 smithi204 ceph-mon[36345]: pgmap v672: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 29 MiB/s wr, 4.59k op/s 2024-03-24T00:47:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:28 smithi123 ceph-mon[31487]: pgmap v672: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 29 MiB/s wr, 4.59k op/s 2024-03-24T00:47:30.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:30 smithi177 ceph-mon[36034]: pgmap v673: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 30 MiB/s wr, 4.89k op/s 2024-03-24T00:47:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:30 smithi204 ceph-mon[36345]: pgmap v673: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 30 MiB/s wr, 4.89k op/s 2024-03-24T00:47:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:30 smithi123 ceph-mon[31487]: pgmap v673: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 30 MiB/s wr, 4.89k op/s 2024-03-24T00:47:32.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:32 smithi177 ceph-mon[36034]: pgmap v674: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 30 MiB/s wr, 5.03k op/s 2024-03-24T00:47:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:32 smithi204 ceph-mon[36345]: pgmap v674: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 30 MiB/s wr, 5.03k op/s 2024-03-24T00:47:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:32 smithi123 ceph-mon[31487]: pgmap v674: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 30 MiB/s wr, 5.03k op/s 2024-03-24T00:47:34.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:34 smithi177 ceph-mon[36034]: pgmap v675: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 29 MiB/s wr, 4.78k op/s 2024-03-24T00:47:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:34 smithi204 ceph-mon[36345]: pgmap v675: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 29 MiB/s wr, 4.78k op/s 2024-03-24T00:47:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:34 smithi123 ceph-mon[31487]: pgmap v675: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 29 MiB/s wr, 4.78k op/s 2024-03-24T00:47:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:36 smithi177 ceph-mon[36034]: pgmap v676: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 33 MiB/s wr, 5.59k op/s 2024-03-24T00:47:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:36 smithi204 ceph-mon[36345]: pgmap v676: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 33 MiB/s wr, 5.59k op/s 2024-03-24T00:47:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:36 smithi123 ceph-mon[31487]: pgmap v676: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 33 MiB/s wr, 5.59k op/s 2024-03-24T00:47:38.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:38 smithi177 ceph-mon[36034]: pgmap v677: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-03-24T00:47:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:38 smithi204 ceph-mon[36345]: pgmap v677: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-03-24T00:47:38.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:38 smithi123 ceph-mon[31487]: pgmap v677: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-03-24T00:47:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:40 smithi204 ceph-mon[36345]: pgmap v678: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 5.35k op/s 2024-03-24T00:47:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:40 smithi123 ceph-mon[31487]: pgmap v678: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 5.35k op/s 2024-03-24T00:47:40.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:40 smithi177 ceph-mon[36034]: pgmap v678: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 5.35k op/s 2024-03-24T00:47:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:42 smithi204 ceph-mon[36345]: pgmap v679: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 28 MiB/s wr, 5.00k op/s 2024-03-24T00:47:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:42 smithi123 ceph-mon[31487]: pgmap v679: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 28 MiB/s wr, 5.00k op/s 2024-03-24T00:47:42.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:42 smithi177 ceph-mon[36034]: pgmap v679: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 28 MiB/s wr, 5.00k op/s 2024-03-24T00:47:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:44 smithi204 ceph-mon[36345]: pgmap v680: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 4.81k op/s 2024-03-24T00:47:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:44 smithi123 ceph-mon[31487]: pgmap v680: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 4.81k op/s 2024-03-24T00:47:44.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:44 smithi177 ceph-mon[36034]: pgmap v680: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 4.81k op/s 2024-03-24T00:47:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.624+0000 7feaae000640 1 -- 172.21.15.123:0/731914596 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5585ad7250c0 con 0x5585ad832f10 2024-03-24T00:47:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.624+0000 7feaae000640 10 client.26506 handle_command_reply: tid=0 2024-03-24T00:47:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.624+0000 7feac9dfc740 1 client.26506 shutdown 2024-03-24T00:47:46.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.625+0000 7f0572a00640 1 -- 172.21.15.123:0/117098499 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56384a6d1790 con 0x56384a77d770 2024-03-24T00:47:46.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.625+0000 7f0572a00640 10 client.26504 handle_command_reply: tid=0 2024-03-24T00:47:46.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.625+0000 7f0596789740 1 client.26504 shutdown 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 msgr2=0x5585ad7d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5585ad7c2060 tx=0x7feaa80052e0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 shutdown_connections 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585ad7c6760 0x5585ad7d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5585ad832f10 0x5585ad835330 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585ad7c15f0 0x5585ad7d0170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 --2- 172.21.15.123:0/731914596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585ad7c0c00 0x5585ad7cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 msgr2=0x56384a7718c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f058002c170 tx=0x7f05800048f0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 >> 172.21.15.123:0/731914596 conn(0x5585ad7b3960 msgr2=0x5585ad7b6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.626+0000 7feaae000640 0 client.26506 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:46.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 shutdown_connections 2024-03-24T00:47:46.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a778e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a77d770 0x56384a77fbd0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.627+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 shutdown_connections 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.626+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a768250 0x56384a771e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0596789740 1 --2- 172.21.15.123:0/117098499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a7718c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 >> 172.21.15.123:0/117098499 conn(0x56384a727200 msgr2=0x56384a759bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0572a00640 0 client.26504 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 shutdown_connections 2024-03-24T00:47:46.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0596789740 1 -- 172.21.15.123:0/117098499 wait complete. 2024-03-24T00:47:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.627+0000 7feac9dfc740 1 -- 172.21.15.123:0/731914596 wait complete. 2024-03-24T00:47:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.628+0000 7feac9dfc740 20 client.26506 trim_cache size 0 max 16384 2024-03-24T00:47:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.627+0000 7f0596789740 20 client.26504 trim_cache size 0 max 16384 2024-03-24T00:47:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:47:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:47:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:47:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:47:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 msgr2=0x7feaa40a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 0x7feaa40a4d30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feab8005c40 tx=0x7feab8009500 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 msgr2=0x7feac01999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feab0000c00 tx=0x7feab002c6f0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 shutdown_connections 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feac01010e0 0x7feac01a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feaa40a2870 0x7feaa40a4d30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feac0100740 0x7feac0199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 --2- 172.21.15.123:0/2051523461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac00ffde0 0x7feac01999d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 >> 172.21.15.123:0/2051523461 conn(0x7feac00fb980 msgr2=0x7feac00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 shutdown_connections 2024-03-24T00:47:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:46.630+0000 7feac5a00640 1 -- 172.21.15.123:0/2051523461 wait complete. 2024-03-24T00:47:46.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.655+0000 7f0596789740 1 Processor -- start 2024-03-24T00:47:46.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.655+0000 7f0596789740 1 -- start start 2024-03-24T00:47:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a784490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c390 con 0x56384a76d3c0 2024-03-24T00:47:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00e950 con 0x56384a768250 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0596789740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c990 con 0x56384a767860 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f058be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0591000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f058be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49822/0 (socket says 172.21.15.123:49822) 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0591000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37156/0 (socket says 172.21.15.123:37156) 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f058be00640 1 -- 172.21.15.123:0/3241404420 learned_addr learned my addr 172.21.15.123:0/3241404420 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.656+0000 7f0591a00640 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a784490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 -- 172.21.15.123:0/3241404420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 msgr2=0x56384a759e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 -- 172.21.15.123:0/3241404420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 msgr2=0x56384a784490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a784490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 -- 172.21.15.123:0/3241404420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56384a3980f0 con 0x56384a768250 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f0591000640 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f058be00640 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0578097d60 tx=0x7f05780ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=331cb9c87960052f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f057d600640 1 -- 172.21.15.123:0/3241404420 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780b8070 con 0x56384a768250 2024-03-24T00:47:46.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f057d600640 1 -- 172.21.15.123:0/3241404420 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f057809b390 con 0x56384a768250 2024-03-24T00:47:46.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.657+0000 7f057d600640 1 -- 172.21.15.123:0/3241404420 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057801c040 con 0x56384a768250 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 -- 172.21.15.123:0/3241404420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 msgr2=0x56384a778e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0578097d60 tx=0x7f05780ce040 comp rx=0 tx=0).stop 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 -- 172.21.15.123:0/3241404420 shutdown_connections 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a76d3c0 0x56384a784490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a768250 0x56384a778e00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 --2- 172.21.15.123:0/3241404420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a767860 0x56384a759e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 -- 172.21.15.123:0/3241404420 >> 172.21.15.123:0/3241404420 conn(0x56384a727200 msgr2=0x56384a769530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 -- 172.21.15.123:0/3241404420 shutdown_connections 2024-03-24T00:47:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.658+0000 7f0596789740 1 -- 172.21.15.123:0/3241404420 wait complete. 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 -- start start 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 Processor -- start 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a776b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a76d3c0 0x56384a777b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00cfd0 con 0x56384a768250 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00e2f0 con 0x56384a767860 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00d630 con 0x56384a76d3c0 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.659+0000 7f0596789740 10 client.0 _ll_register_callbacks cb 0x7f0580009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49838/0 (socket says 172.21.15.123:49838) 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 -- 172.21.15.123:0/4019421183 learned_addr learned my addr 172.21.15.123:0/4019421183 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:46.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591a00640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a76d3c0 0x56384a777b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f058be00640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a776b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 -- 172.21.15.123:0/4019421183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a76d3c0 msgr2=0x56384a777b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a76d3c0 0x56384a777b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 -- 172.21.15.123:0/4019421183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 msgr2=0x56384a776b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a776b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 -- 172.21.15.123:0/4019421183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56384a778170 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0591000640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0580003f20 tx=0x7f0580057000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99e4bf1109e75750 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.660+0000 7f0572a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058006a070 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f058004bba0 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f057809b390 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0580004490 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0596789740 10 client.26510 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56384a7d91c0 con 0x56384a767860 2024-03-24T00:47:46.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f057809b390 con 0x56384a767860 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.661+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0580004f60 con 0x56384a767860 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 10 client.26510 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 10 client.26510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 20 client.26510 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a7d9ae0 0x56384a7dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 4 client.26510 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56384a6cbae0 con 0x56384a7d9ae0 2024-03-24T00:47:46.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f058be00640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a7d9ae0 0x56384a7dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f058be00640 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a7d9ae0 0x56384a7dbf00 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d6413893507489 server_cookie=7f9f25d7d4604397 in_seq=0 out_seq=0 2024-03-24T00:47:46.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.662+0000 7f0572a00640 10 client.26510 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:46.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.663+0000 7f0572a00640 1 -- 172.21.15.123:0/4019421183 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56384a6cbae0 con 0x56384a7d9ae0 2024-03-24T00:47:46.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.663+0000 7f0572a00640 10 client.26510 handle_command_reply: tid=0 2024-03-24T00:47:46.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.663+0000 7f0596789740 1 client.26510 shutdown 2024-03-24T00:47:46.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.663+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 msgr2=0x56384a776630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.663+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0580003f20 tx=0x7f0580057000 comp rx=0 tx=0).stop 2024-03-24T00:47:46.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 shutdown_connections 2024-03-24T00:47:46.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56384a7d9ae0 0x56384a7dbf00 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56384a76d3c0 0x56384a777b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56384a768250 0x56384a776b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 --2- 172.21.15.123:0/4019421183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56384a767860 0x56384a776630 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0572a00640 0 client.26510 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:46.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 >> 172.21.15.123:0/4019421183 conn(0x56384a727200 msgr2=0x56384a766410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 shutdown_connections 2024-03-24T00:47:46.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 1 -- 172.21.15.123:0/4019421183 wait complete. 2024-03-24T00:47:46.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.664+0000 7f0596789740 20 client.26510 trim_cache size 0 max 16384 2024-03-24T00:47:46.672 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:47:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:47:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:47:46.672 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 msgr2=0x7f05740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 0x7f05740a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f058400fe40 tx=0x7f0584014000 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 msgr2=0x7f058c19a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f057800e9f0 tx=0x7f057800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 shutdown_connections 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f058c103150 0x7f058c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f058c102790 0x7f058c19a0e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 --2- 172.21.15.123:0/2607431526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f058c1015a0 0x7f058c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.665+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 >> 172.21.15.123:0/2607431526 conn(0x7f058c0fcb10 msgr2=0x7f058c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.666+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 shutdown_connections 2024-03-24T00:47:46.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.666+0000 7f0592400640 1 -- 172.21.15.123:0/2607431526 wait complete. 2024-03-24T00:47:46.688 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:47:46.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:46.754 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:46 smithi123 ceph-mon[31487]: pgmap v681: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 5.34k op/s 2024-03-24T00:47:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:46 smithi204 ceph-mon[36345]: pgmap v681: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 5.34k op/s 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 -- 172.21.15.123:0/3304930661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 msgr2=0x7f340c102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 --2- 172.21.15.123:0/3304930661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c102b70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009900 tx=0x7f33f802bc30 comp rx=0 tx=0).stop 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 -- 172.21.15.123:0/3304930661 shutdown_connections 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 --2- 172.21.15.123:0/3304930661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 0x7f340c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 --2- 172.21.15.123:0/3304930661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c102b70 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 --2- 172.21.15.123:0/3304930661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 -- 172.21.15.123:0/3304930661 >> 172.21.15.123:0/3304930661 conn(0x7f340c0fca70 msgr2=0x7f340c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.882+0000 7f3411a00640 1 -- 172.21.15.123:0/3304930661 shutdown_connections 2024-03-24T00:47:46.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.883+0000 7f3411a00640 1 -- 172.21.15.123:0/3304930661 wait complete. 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.883+0000 7f3411a00640 1 Processor -- start 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 -- start start 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 0x7f340c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c10d1e0 con 0x7f340c101500 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c10d060 con 0x7f340c1026f0 2024-03-24T00:47:46.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c10d360 con 0x7f340c1030b0 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f340be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f340b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f340be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39848/0 (socket says 172.21.15.123:39848) 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f340be00640 1 -- 172.21.15.123:0/1916213746 learned_addr learned my addr 172.21.15.123:0/1916213746 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.884+0000 7f3411000640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 0x7f340c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 -- 172.21.15.123:0/1916213746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 msgr2=0x7f340c1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 0x7f340c1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 -- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 msgr2=0x7f340c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 -- 172.21.15.123:0/1916213746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c1a1640 con 0x7f340c1026f0 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340be00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:46.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.885+0000 7f340b400640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c199f60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f33f8036040 tx=0x7f33f802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:46.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.886+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8043020 con 0x7f340c1026f0 2024-03-24T00:47:46.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.886+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f80054e0 con 0x7f340c1026f0 2024-03-24T00:47:46.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.886+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c1a18d0 con 0x7f340c1026f0 2024-03-24T00:47:46.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.886+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8003c30 con 0x7f340c1026f0 2024-03-24T00:47:46.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.886+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c1a1e40 con 0x7f340c1026f0 2024-03-24T00:47:46.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.887+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c1a2010 con 0x7f340c1026f0 2024-03-24T00:47:46.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.889+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f33f8046020 con 0x7f340c1026f0 2024-03-24T00:47:46.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.890+0000 7f3408c00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 0x7f33f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.891+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f33f80f1030 con 0x7f340c1026f0 2024-03-24T00:47:46.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.891+0000 7f340be00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 0x7f33f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:46.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.891+0000 7f340be00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 0x7f33f00a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0058c0 tx=0x7f33fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:46.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:46.892+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33f804d020 con 0x7f340c1026f0 2024-03-24T00:47:46.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:46 smithi177 ceph-mon[36034]: pgmap v681: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 5.34k op/s 2024-03-24T00:47:47.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.186+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f340c1a2300 con 0x7f340c1026f0 2024-03-24T00:47:47.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.187+0000 7f3408c00640 1 -- 172.21.15.123:0/1916213746 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f33f80be0e0 con 0x7f340c1026f0 2024-03-24T00:47:47.188 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:47:47.189 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:47:47.189 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:47:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 msgr2=0x7f33f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 0x7f33f00a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0058c0 tx=0x7f33fc005f90 comp rx=0 tx=0).stop 2024-03-24T00:47:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 msgr2=0x7f340c199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c199f60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f33f8036040 tx=0x7f33f802c650 comp rx=0 tx=0).stop 2024-03-24T00:47:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 shutdown_connections 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f340c1030b0 0x7f340c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340c1026f0 0x7f340c199f60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f33f00a2880 0x7f33f00a4d40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 --2- 172.21.15.123:0/1916213746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f340c101500 0x7f340c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 >> 172.21.15.123:0/1916213746 conn(0x7f340c0fca70 msgr2=0x7f340c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 shutdown_connections 2024-03-24T00:47:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.190+0000 7f3411a00640 1 -- 172.21.15.123:0/1916213746 wait complete. 2024-03-24T00:47:47.211 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:47:47.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.425+0000 7f5b5f200640 1 -- 172.21.15.123:0/4255072223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 msgr2=0x7f5b5810b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.425+0000 7f5b5f200640 1 --2- 172.21.15.123:0/4255072223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b5810b9e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009ca0 tx=0x7f5b5002c180 comp rx=0 tx=0).stop 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 -- 172.21.15.123:0/4255072223 shutdown_connections 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 --2- 172.21.15.123:0/4255072223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b5810b9e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 --2- 172.21.15.123:0/4255072223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 0x7f5b58074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 --2- 172.21.15.123:0/4255072223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b58073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 -- 172.21.15.123:0/4255072223 >> 172.21.15.123:0/4255072223 conn(0x7f5b580fcc40 msgr2=0x7f5b580ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 -- 172.21.15.123:0/4255072223 shutdown_connections 2024-03-24T00:47:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.426+0000 7f5b5f200640 1 -- 172.21.15.123:0/4255072223 wait complete. 2024-03-24T00:47:47.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 Processor -- start 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 -- start start 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 0x7f5b58199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b5819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b581a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5810f530 con 0x7f5b580740a0 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5810f3b0 con 0x7f5b58109540 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.427+0000 7f5b5f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5810f6b0 con 0x7f5b58075680 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b57400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b5819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b581a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b57400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b5819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37196/0 (socket says 172.21.15.123:37196) 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b57400640 1 -- 172.21.15.123:0/2100687189 learned_addr learned my addr 172.21.15.123:0/2100687189 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b57e00640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 0x7f5b58199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 -- 172.21.15.123:0/2100687189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 msgr2=0x7f5b5819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b5819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 -- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 msgr2=0x7f5b58199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 0x7f5b58199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.428+0000 7f5b5d200640 1 -- 172.21.15.123:0/2100687189 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b581a1790 con 0x7f5b58109540 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.429+0000 7f5b5d200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b581a1120 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5b50037040 tx=0x7f5b500093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.429+0000 7f5b54c00640 1 -- 172.21.15.123:0/2100687189 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50039040 con 0x7f5b58109540 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.429+0000 7f5b54c00640 1 -- 172.21.15.123:0/2100687189 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b50044070 con 0x7f5b58109540 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.429+0000 7f5b54c00640 1 -- 172.21.15.123:0/2100687189 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500042a0 con 0x7f5b58109540 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.430+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b581a1a20 con 0x7f5b58109540 2024-03-24T00:47:47.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.430+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b581a1e90 con 0x7f5b58109540 2024-03-24T00:47:47.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.433+0000 7f5b54c00640 1 -- 172.21.15.123:0/2100687189 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5b50004440 con 0x7f5b58109540 2024-03-24T00:47:47.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.433+0000 7f5b61ff7740 1 Processor -- start 2024-03-24T00:47:47.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 -- start start 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f234810 0x55987f234c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f2363a0 0x55987f23cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f1992c0 con 0x55987f2363a0 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f141c90 con 0x55987f234810 2024-03-24T00:47:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f195360 con 0x55987f2359e0 2024-03-24T00:47:47.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f234810 0x55987f234c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37210/0 (socket says 172.21.15.123:37210) 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b57e00640 1 -- 172.21.15.123:0/3996780703 learned_addr learned my addr 172.21.15.123:0/3996780703 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.434+0000 7f5b54c00640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 0x7f5b3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57400640 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f2363a0 0x55987f23cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b54c00640 1 -- 172.21.15.123:0/2100687189 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5b500f0cb0 con 0x7f5b58109540 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 -- 172.21.15.123:0/3996780703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f234810 msgr2=0x55987f234c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f234810 0x55987f234c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 -- 172.21.15.123:0/3996780703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f2363a0 msgr2=0x55987f23cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f2363a0 0x55987f23cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b5d200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 0x7f5b3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 -- 172.21.15.123:0/3996780703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55987f166080 con 0x55987f2359e0 2024-03-24T00:47:47.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.435+0000 7f5b57e00640 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009d20 tx=0x7f5b48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3252291e16caf396 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.436+0000 7f5b4d600640 1 -- 172.21.15.123:0/3996780703 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4801e070 con 0x55987f2359e0 2024-03-24T00:47:47.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.436+0000 7f5b4d600640 1 -- 172.21.15.123:0/3996780703 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b48019b10 con 0x55987f2359e0 2024-03-24T00:47:47.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.436+0000 7f5b4d600640 1 -- 172.21.15.123:0/3996780703 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800eb80 con 0x55987f2359e0 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- 172.21.15.123:0/3996780703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 msgr2=0x55987f235e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009d20 tx=0x7f5b48011810 comp rx=0 tx=0).stop 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b5d200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 0x7f5b3c0a4d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b500324c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- 172.21.15.123:0/3996780703 shutdown_connections 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f2363a0 0x55987f23cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f235e60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/3996780703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f234810 0x55987f234c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- 172.21.15.123:0/3996780703 >> 172.21.15.123:0/3996780703 conn(0x55987f22fc00 msgr2=0x55987f230000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- 172.21.15.123:0/3996780703 shutdown_connections 2024-03-24T00:47:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- 172.21.15.123:0/3996780703 wait complete. 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.437+0000 7f5b61ff7740 1 -- start start 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 Processor -- start 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 0x55987f23ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2363a0 0x55987f246010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f23d3b0 con 0x55987f234810 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f1550d0 con 0x55987f2363a0 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987f23d530 con 0x55987f2359e0 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b61ff7740 10 client.0 _ll_register_callbacks cb 0x55987f159e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37216/0 (socket says 172.21.15.123:37216) 2024-03-24T00:47:47.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.438+0000 7f5b57e00640 1 -- 172.21.15.123:0/1644849077 learned_addr learned my addr 172.21.15.123:0/1644849077 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57400640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 0x55987f23ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57400640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2363a0 0x55987f246010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 -- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2363a0 msgr2=0x55987f246010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2363a0 0x55987f246010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 -- 172.21.15.123:0/1644849077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 msgr2=0x55987f23ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 0x55987f23ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 -- 172.21.15.123:0/1644849077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55987eead3f0 con 0x55987f2359e0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57400640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 0x55987f23ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b57e00640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005c50 tx=0x7f5b48005c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65db224b13e5fb67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.439+0000 7f5b42000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48026070 con 0x55987f2359e0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5b48019840 con 0x55987f2359e0 2024-03-24T00:47:47.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b48019b10 con 0x55987f2359e0 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b48032050 con 0x55987f2359e0 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b61ff7740 10 client.26539 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55987ef777d0 con 0x55987f2359e0 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.440+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b48019b10 con 0x55987f2359e0 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5b4800ecf0 con 0x55987f2359e0 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 10 client.26539 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 10 client.26539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 20 client.26539 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f24a170 0x55987f24c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 4 client.26539 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55987f1a6800 con 0x55987f24a170 2024-03-24T00:47:47.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.441+0000 7f5b57400640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f24a170 0x55987f24c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.442+0000 7f5b57400640 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f24a170 0x55987f24c5a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=458ad8ed27254970 server_cookie=7af5e275ddf79e7b in_seq=0 out_seq=0 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.442+0000 7f5b42000640 10 client.26539 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b42000640 1 -- 172.21.15.123:0/1644849077 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55987f1a6800 con 0x55987f24a170 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b42000640 10 client.26539 handle_command_reply: tid=0 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 client.26539 shutdown 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 msgr2=0x55987f23efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005c50 tx=0x7f5b48005c80 comp rx=0 tx=0).stop 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 shutdown_connections 2024-03-24T00:47:47.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2363a0 0x55987f246010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f24a170 0x55987f24c5a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f2359e0 0x55987f23efd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/1644849077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f234810 0x55987f23ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 >> 172.21.15.123:0/1644849077 conn(0x55987f22fc00 msgr2=0x55987f22eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b42000640 0 client.26539 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.443+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 shutdown_connections 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.444+0000 7f5b61ff7740 1 -- 172.21.15.123:0/1644849077 wait complete. 2024-03-24T00:47:47.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.444+0000 7f5b61ff7740 20 client.26539 trim_cache size 0 max 16384 2024-03-24T00:47:47.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 Processor -- start 2024-03-24T00:47:47.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 -- start start 2024-03-24T00:47:47.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f23c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f245fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f238a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581a8c60 con 0x55987f22fc00 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581ab220 con 0x55987f2359e0 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.463+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581a9260 con 0x55987f234810 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f23c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f245fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f245fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37232/0 (socket says 172.21.15.123:37232) 2024-03-24T00:47:47.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57e00640 1 -- 172.21.15.123:0/2218742946 learned_addr learned my addr 172.21.15.123:0/2218742946 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f238a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 -- 172.21.15.123:0/2218742946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 msgr2=0x55987f245fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f245fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 -- 172.21.15.123:0/2218742946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 msgr2=0x55987f238a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f238a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.464+0000 7f5b57400640 1 -- 172.21.15.123:0/2218742946 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55987f24cc20 con 0x55987f22fc00 2024-03-24T00:47:47.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.465+0000 7f5b57400640 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f23c8c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55987f24ce20 tx=0x7f5b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28645db788f8aa46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.465+0000 7f5b4cc00640 1 -- 172.21.15.123:0/2218742946 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b440250d0 con 0x55987f22fc00 2024-03-24T00:47:47.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.465+0000 7f5b4cc00640 1 -- 172.21.15.123:0/2218742946 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b44025270 con 0x55987f22fc00 2024-03-24T00:47:47.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.465+0000 7f5b4cc00640 1 -- 172.21.15.123:0/2218742946 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44004dd0 con 0x55987f22fc00 2024-03-24T00:47:47.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 -- 172.21.15.123:0/2218742946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 msgr2=0x55987f23c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f23c8c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55987f24ce20 tx=0x7f5b440046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 -- 172.21.15.123:0/2218742946 shutdown_connections 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f238a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f245fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/2218742946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f23c8c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.466+0000 7f5b61ff7740 1 -- 172.21.15.123:0/2218742946 >> 172.21.15.123:0/2218742946 conn(0x55987f23dca0 msgr2=0x55987f2336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.467+0000 7f5b61ff7740 1 -- 172.21.15.123:0/2218742946 shutdown_connections 2024-03-24T00:47:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.467+0000 7f5b61ff7740 1 -- 172.21.15.123:0/2218742946 wait complete. 2024-03-24T00:47:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.467+0000 7f5b61ff7740 1 -- start start 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.467+0000 7f5b61ff7740 1 Processor -- start 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f242720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f2456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581a98a0 con 0x55987f22fc00 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581aabc0 con 0x55987f2359e0 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581a9f00 con 0x55987f234810 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b61ff7740 10 client.0 _ll_register_callbacks cb 0x7f5b4801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.468+0000 7f5b57400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39908/0 (socket says 172.21.15.123:39908) 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 -- 172.21.15.123:0/673174373 learned_addr learned my addr 172.21.15.123:0/673174373 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f2456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57e00640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f242720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 -- 172.21.15.123:0/673174373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 msgr2=0x55987f242720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f242720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 -- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 msgr2=0x55987f2456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f2456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57400640 1 -- 172.21.15.123:0/673174373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55987f23fbc0 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.469+0000 7f5b57e00640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f242720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b57400640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b44011840 tx=0x7f5b44005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f63736196caf297 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b42000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44036060 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b44025270 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5b44004230 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b44004430 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b61ff7740 10 client.16362 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.470+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55987f2a6360 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b44025270 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5b44030030 con 0x55987f22fc00 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 10 client.16362 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 20 client.16362 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f2363a0 0x55987f2a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 4 client.16362 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55987f1a0b50 con 0x55987f2363a0 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.471+0000 7f5b57e00640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f2363a0 0x55987f2a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b57e00640 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f2363a0 0x55987f2a8c70 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eccc562d0336aec server_cookie=9fa2498f30902145 in_seq=0 out_seq=0 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b42000640 10 client.16362 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b42000640 1 -- 172.21.15.123:0/673174373 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55987f1a0b50 con 0x55987f2363a0 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b42000640 10 client.16362 handle_command_reply: tid=0 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 client.16362 shutdown 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 msgr2=0x55987f2421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b44011840 tx=0x7f5b44005f90 comp rx=0 tx=0).stop 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 shutdown_connections 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55987f2359e0 0x55987f2456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55987f2363a0 0x55987f2a8c70 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55987f234810 0x55987f242720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 --2- 172.21.15.123:0/673174373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55987f22fc00 0x55987f2421e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b42000640 0 client.16362 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 >> 172.21.15.123:0/673174373 conn(0x55987f23dca0 msgr2=0x55987f2336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 shutdown_connections 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.472+0000 7f5b61ff7740 1 -- 172.21.15.123:0/673174373 wait complete. 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b61ff7740 20 client.16362 trim_cache size 0 max 16384 2024-03-24T00:47:47.476 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 msgr2=0x7f5b3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 0x7f5b3c0a4d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b500324c0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 msgr2=0x7f5b581a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b581a1120 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5b50037040 tx=0x7f5b500093d0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 shutdown_connections 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58109540 0x7f5b581a1120 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b58075680 0x7f5b5819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f5b3c0a2880 0x7f5b3c0a4d40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 --2- 172.21.15.123:0/2100687189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b580740a0 0x7f5b58199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:47.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.473+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 >> 172.21.15.123:0/2100687189 conn(0x7f5b580fcc40 msgr2=0x7f5b581054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:47.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.474+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 shutdown_connections 2024-03-24T00:47:47.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:47.474+0000 7f5b5f200640 1 -- 172.21.15.123:0/2100687189 wait complete. 2024-03-24T00:47:47.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:47 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1916213746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:47:47.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1916213746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:47:47.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:47 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1916213746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:47:48.494 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:47:48.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:48.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:47:48.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 -- 172.21.15.123:0/2024865108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc103130 msgr2=0x7f8dbc1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 --2- 172.21.15.123:0/2024865108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc103130 0x7f8dbc1099c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009ca0 tx=0x7f8db402c100 comp rx=0 tx=0).stop 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 -- 172.21.15.123:0/2024865108 shutdown_connections 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 --2- 172.21.15.123:0/2024865108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc103130 0x7f8dbc1099c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 --2- 172.21.15.123:0/2024865108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc102770 0x7f8dbc102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 --2- 172.21.15.123:0/2024865108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc101580 0x7f8dbc1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 -- 172.21.15.123:0/2024865108 >> 172.21.15.123:0/2024865108 conn(0x7f8dbc0fcb10 msgr2=0x7f8dbc0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 -- 172.21.15.123:0/2024865108 shutdown_connections 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.722+0000 7f8dbb200640 1 -- 172.21.15.123:0/2024865108 wait complete. 2024-03-24T00:47:48.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.724+0000 7f8dbb200640 1 Processor -- start 2024-03-24T00:47:48.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.724+0000 7f8dbb200640 1 -- start start 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 0x7f8dbc199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 0x7f8dbc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 0x7f8dbc1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc10d200 con 0x7f8dbc101580 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc10d080 con 0x7f8dbc102770 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dbb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc10d380 con 0x7f8dbc103130 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 0x7f8dbc199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 0x7f8dbc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 0x7f8dbc1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 0x7f8dbc199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49928/0 (socket says 172.21.15.123:49928) 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 0x7f8dbc199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39928/0 (socket says 172.21.15.123:39928) 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8db3e00640 1 -- 172.21.15.123:0/845638894 learned_addr learned my addr 172.21.15.123:0/845638894 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 -- 172.21.15.123:0/845638894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 msgr2=0x7f8dbc199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 0x7f8dbc199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 -- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 msgr2=0x7f8dbc199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 0x7f8dbc199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.725+0000 7f8dba800640 1 -- 172.21.15.123:0/845638894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc1a1620 con 0x7f8dbc103130 2024-03-24T00:47:48.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.726+0000 7f8dba800640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 0x7f8dbc1a0fb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8db4037040 tx=0x7f8db4035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.726+0000 7f8db2a00640 1 -- 172.21.15.123:0/845638894 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4039040 con 0x7f8dbc103130 2024-03-24T00:47:48.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.727+0000 7f8db2a00640 1 -- 172.21.15.123:0/845638894 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8db4044070 con 0x7f8dbc103130 2024-03-24T00:47:48.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.727+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dbc1a18b0 con 0x7f8dbc103130 2024-03-24T00:47:48.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.727+0000 7f8db2a00640 1 -- 172.21.15.123:0/845638894 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4004330 con 0x7f8dbc103130 2024-03-24T00:47:48.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.727+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dbc1a1dc0 con 0x7f8dbc103130 2024-03-24T00:47:48.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.731+0000 7f8db2a00640 1 -- 172.21.15.123:0/845638894 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8db4003970 con 0x7f8dbc103130 2024-03-24T00:47:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.732+0000 7f8dc3668740 1 Processor -- start 2024-03-24T00:47:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.732+0000 7f8dc3668740 1 -- start start 2024-03-24T00:47:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.733+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.733+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.733+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39936/0 (socket says 172.21.15.123:39936) 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.733+0000 7f8db3e00640 1 -- 172.21.15.123:0/1698403822 learned_addr learned my addr 172.21.15.123:0/1698403822 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.733+0000 7f8db2a00640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 0x7f8da00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db2a00640 1 -- 172.21.15.123:0/845638894 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8db40f0e80 con 0x7f8dbc103130 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca8d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fc9eaeb0 con 0x55d0fca852b0 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fc993a60 con 0x55d0fca8ae10 2024-03-24T00:47:48.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fc9e6f50 con 0x55d0fca85ca0 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8dba800640 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca8d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 -- 172.21.15.123:0/1698403822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 msgr2=0x55d0fca8a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 -- 172.21.15.123:0/1698403822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 msgr2=0x55d0fca8d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca8d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.734+0000 7f8db3e00640 1 -- 172.21.15.123:0/1698403822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fc7c77e0 con 0x55d0fca852b0 2024-03-24T00:47:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.735+0000 7f8db3e00640 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8da40155c0 tx=0x7f8da4009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca47c3af32109a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.735+0000 7f8db9e00640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 0x7f8da00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.735+0000 7f8dab000640 1 -- 172.21.15.123:0/1698403822 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da40150d0 con 0x55d0fca852b0 2024-03-24T00:47:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.735+0000 7f8dab000640 1 -- 172.21.15.123:0/1698403822 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8da4015270 con 0x55d0fca852b0 2024-03-24T00:47:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.735+0000 7f8dab000640 1 -- 172.21.15.123:0/1698403822 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da400fac0 con 0x55d0fca852b0 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 msgr2=0x55d0fca856d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8db9e00640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 0x7f8da00a4d40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc1025d0 tx=0x7f8dac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8da40155c0 tx=0x7f8da4009950 comp rx=0 tx=0).stop 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 shutdown_connections 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca8d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 --2- 172.21.15.123:0/1698403822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca856d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 >> 172.21.15.123:0/1698403822 conn(0x55d0fca80820 msgr2=0x55d0fca80c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 shutdown_connections 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.736+0000 7f8dc3668740 1 -- 172.21.15.123:0/1698403822 wait complete. 2024-03-24T00:47:48.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.737+0000 7f8dc3668740 1 -- start start 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.737+0000 7f8dc3668740 1 Processor -- start 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca8f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca96a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fca8ddc0 con 0x55d0fca852b0 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fc9a6eb0 con 0x55d0fca8ae10 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fca8df40 con 0x55d0fca85ca0 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.738+0000 7f8dc3668740 10 client.0 _ll_register_callbacks cb 0x55d0fc99b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37272/0 (socket says 172.21.15.123:37272) 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca96a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8db3e00640 1 -- 172.21.15.123:0/3496909167 learned_addr learned my addr 172.21.15.123:0/3496909167 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 -- 172.21.15.123:0/3496909167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 msgr2=0x55d0fca8f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 -- 172.21.15.123:0/3496909167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 msgr2=0x55d0fca8f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca8f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 -- 172.21.15.123:0/3496909167 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0fc827f70 con 0x55d0fca8ae10 2024-03-24T00:47:48.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8db3e00640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.739+0000 7f8dba800640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca96a20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009ca0 tx=0x7f8db40bd6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41181a42266ab3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8d9f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40098d0 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8da4015270 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8db40bda90 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8db40bdf30 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8dc3668740 10 client.26546 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d0fc6bf170 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.740+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8da4015270 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8db403c030 con 0x55d0fca8ae10 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 10 client.26546 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:48.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 20 client.26546 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fca9b280 0x55d0fca9d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 4 client.26546 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0fc9f83f0 con 0x55d0fca9b280 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.741+0000 7f8db9e00640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fca9b280 0x55d0fca9d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.742+0000 7f8db9e00640 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fca9b280 0x55d0fca9d720 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe198ea40764fe2f server_cookie=4ab2ea5da5a6eabf in_seq=0 out_seq=0 2024-03-24T00:47:48.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.742+0000 7f8d9f400640 10 client.26546 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.742+0000 7f8d9f400640 1 -- 172.21.15.123:0/3496909167 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d0fc9f83f0 con 0x55d0fca9b280 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.742+0000 7f8d9f400640 10 client.26546 handle_command_reply: tid=0 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 client.26546 shutdown 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 msgr2=0x55d0fca96a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca96a20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009ca0 tx=0x7f8db40bd6c0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 shutdown_connections 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca96a20 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fca9b280 0x55d0fca9d720 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca85ca0 0x55d0fca8f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8d9f400640 0 client.26546 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 --2- 172.21.15.123:0/3496909167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca852b0 0x55d0fca8f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 >> 172.21.15.123:0/3496909167 conn(0x55d0fca80820 msgr2=0x55d0fca7fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 shutdown_connections 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 1 -- 172.21.15.123:0/3496909167 wait complete. 2024-03-24T00:47:48.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.743+0000 7f8dc3668740 20 client.26546 trim_cache size 0 max 16384 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 Processor -- start 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 -- start start 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca840f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 0x55d0fca969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca8ae10 0x55d0fcaa2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1a8a10 con 0x55d0fca8ae10 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1aafd0 con 0x55d0fca85ca0 2024-03-24T00:47:48.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.770+0000 7f8dc3668740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1a9010 con 0x55d0fca852b0 2024-03-24T00:47:48.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca840f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca840f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37286/0 (socket says 172.21.15.123:37286) 2024-03-24T00:47:48.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db9e00640 1 -- 172.21.15.123:0/842611050 learned_addr learned my addr 172.21.15.123:0/842611050 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:48.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 0x55d0fca969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8dba800640 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca8ae10 0x55d0fcaa2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 -- 172.21.15.123:0/842611050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 msgr2=0x55d0fca840f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca840f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 -- 172.21.15.123:0/842611050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca8ae10 msgr2=0x55d0fcaa2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca8ae10 0x55d0fcaa2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.771+0000 7f8db3e00640 1 -- 172.21.15.123:0/842611050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fca9dda0 con 0x55d0fca85ca0 2024-03-24T00:47:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.772+0000 7f8db3e00640 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 0x55d0fca969e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8da4016fd0 tx=0x7f8da4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e218aca3fb7d8d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.772+0000 7f8daa600640 1 -- 172.21.15.123:0/842611050 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da40150d0 con 0x55d0fca85ca0 2024-03-24T00:47:48.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.772+0000 7f8daa600640 1 -- 172.21.15.123:0/842611050 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8da4015270 con 0x55d0fca85ca0 2024-03-24T00:47:48.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.772+0000 7f8daa600640 1 -- 172.21.15.123:0/842611050 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da40150d0 con 0x55d0fca85ca0 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.773+0000 7f8dc3668740 1 -- 172.21.15.123:0/842611050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 msgr2=0x55d0fca969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.773+0000 7f8dc3668740 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 0x55d0fca969e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8da4016fd0 tx=0x7f8da4009780 comp rx=0 tx=0).stop 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 -- 172.21.15.123:0/842611050 shutdown_connections 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca8ae10 0x55d0fcaa2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca85ca0 0x55d0fca969e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 --2- 172.21.15.123:0/842611050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca840f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 -- 172.21.15.123:0/842611050 >> 172.21.15.123:0/842611050 conn(0x55d0fca80820 msgr2=0x55d0fca7fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 -- 172.21.15.123:0/842611050 shutdown_connections 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 -- 172.21.15.123:0/842611050 wait complete. 2024-03-24T00:47:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.774+0000 7f8dc3668740 1 -- start start 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 Processor -- start 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 0x55d0fca95770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca95cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1a9650 con 0x55d0fca85ca0 2024-03-24T00:47:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1aa970 con 0x55d0fca8ae10 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1a9cb0 con 0x55d0fca852b0 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8dc3668740 10 client.0 _ll_register_callbacks cb 0x7f8db40b9410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.775+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 0x55d0fca95770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37302/0 (socket says 172.21.15.123:37302) 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 -- 172.21.15.123:0/830460530 learned_addr learned my addr 172.21.15.123:0/830460530 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8dba800640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca95cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 -- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 msgr2=0x55d0fca95cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca95cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 -- 172.21.15.123:0/830460530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 msgr2=0x55d0fca95770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 0x55d0fca95770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db9e00640 1 -- 172.21.15.123:0/830460530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0fca81290 con 0x55d0fca852b0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.776+0000 7f8db3e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 0x55d0fca95770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8db9e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55d0fca9dfd0 tx=0x7f8dac014070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e164fc8b546fda83 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8d9f400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac004220 con 0x55d0fca852b0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8dac0043c0 con 0x55d0fca852b0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8da4015270 con 0x55d0fca852b0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dac0035f0 con 0x55d0fca852b0 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8dc3668740 10 client.26563 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:48.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.777+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d0fca81590 con 0x55d0fca852b0 2024-03-24T00:47:48.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8da4015270 con 0x55d0fca852b0 2024-03-24T00:47:48.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8dac012850 con 0x55d0fca852b0 2024-03-24T00:47:48.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 10 client.26563 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 10 client.26563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 20 client.26563 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fcb00ea0 0x55d0fcb032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 4 client.26563 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d0fc9f2740 con 0x55d0fcb00ea0 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.778+0000 7f8db3e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fcb00ea0 0x55d0fcb032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.779+0000 7f8d9f400640 10 client.26563 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:48.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.779+0000 7f8db3e00640 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fcb00ea0 0x55d0fcb032c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70a9eb78049e347f server_cookie=19d6b564d6e37a5a in_seq=0 out_seq=0 2024-03-24T00:47:48.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8d9f400640 1 -- 172.21.15.123:0/830460530 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d0fc9f2740 con 0x55d0fcb00ea0 2024-03-24T00:47:48.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8d9f400640 10 client.26563 handle_command_reply: tid=0 2024-03-24T00:47:48.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 client.26563 shutdown 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 msgr2=0x55d0fca95230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55d0fca9dfd0 tx=0x7f8dac014070 comp rx=0 tx=0).stop 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 shutdown_connections 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0fca8ae10 0x55d0fca95cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d0fca85ca0 0x55d0fca95770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d0fcb00ea0 0x55d0fcb032c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 --2- 172.21.15.123:0/830460530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d0fca852b0 0x55d0fca95230 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8d9f400640 0 client.26563 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 >> 172.21.15.123:0/830460530 conn(0x55d0fca80820 msgr2=0x55d0fca866a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 shutdown_connections 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 1 -- 172.21.15.123:0/830460530 wait complete. 2024-03-24T00:47:48.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.781+0000 7f8dc3668740 20 client.26563 trim_cache size 0 max 16384 2024-03-24T00:47:48.784 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:47:48.784 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:47:48.784 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "77aff907-ed5f-41b8-92a8-b4b53aaa4a28", 2024-03-24T00:47:48.784 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:47:48.784 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.782+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 msgr2=0x7f8da00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.782+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 0x7f8da00a4d40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc1025d0 tx=0x7f8dac005f90 comp rx=0 tx=0).stop 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.782+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 msgr2=0x7f8dbc1a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.782+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 0x7f8dbc1a0fb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8db4037040 tx=0x7f8db4035d10 comp rx=0 tx=0).stop 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 shutdown_connections 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dbc103130 0x7f8dbc1a0fb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dbc102770 0x7f8dbc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8da00a2880 0x7f8da00a4d40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 --2- 172.21.15.123:0/845638894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc101580 0x7f8dbc199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:48.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 >> 172.21.15.123:0/845638894 conn(0x7f8dbc0fcb10 msgr2=0x7f8dbc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:48.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 shutdown_connections 2024-03-24T00:47:48.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:48.783+0000 7f8dbb200640 1 -- 172.21.15.123:0/845638894 wait complete. 2024-03-24T00:47:48.800 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:48 smithi123 ceph-mon[31487]: pgmap v682: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 27 MiB/s wr, 4.88k op/s 2024-03-24T00:47:48.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:47:48.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:47:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:48 smithi204 ceph-mon[36345]: pgmap v682: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 27 MiB/s wr, 4.88k op/s 2024-03-24T00:47:48.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:48 smithi177 ceph-mon[36034]: pgmap v682: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 27 MiB/s wr, 4.88k op/s 2024-03-24T00:47:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 -- 172.21.15.123:0/1119347537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178077c40 msgr2=0x7f41780784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 --2- 172.21.15.123:0/1119347537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178077c40 0x7f41780784e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4170009ca0 tx=0x7f417002c100 comp rx=0 tx=0).stop 2024-03-24T00:47:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 -- 172.21.15.123:0/1119347537 shutdown_connections 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 --2- 172.21.15.123:0/1119347537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178077c40 0x7f41780784e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 --2- 172.21.15.123:0/1119347537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41780772a0 0x7f4178077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 --2- 172.21.15.123:0/1119347537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4178076060 0x7f4178076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 -- 172.21.15.123:0/1119347537 >> 172.21.15.123:0/1119347537 conn(0x7f41780fed10 msgr2=0x7f4178101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.015+0000 7f417dc00640 1 -- 172.21.15.123:0/1119347537 shutdown_connections 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.016+0000 7f417dc00640 1 -- 172.21.15.123:0/1119347537 wait complete. 2024-03-24T00:47:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 Processor -- start 2024-03-24T00:47:49.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 -- start start 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4178077c40 0x7f41781a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178113830 con 0x7f4178077c40 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781136b0 con 0x7f41780772a0 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.017+0000 7f417dc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781139b0 con 0x7f4178076060 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:49988/0 (socket says 172.21.15.123:49988) 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37334/0 (socket says 172.21.15.123:37334) 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177400640 1 -- 172.21.15.123:0/1642024779 learned_addr learned my addr 172.21.15.123:0/1642024779 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f417d200640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4178077c40 0x7f41781a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 -- 172.21.15.123:0/1642024779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 msgr2=0x7f417819e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 -- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4178077c40 msgr2=0x7f41781a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4178077c40 0x7f41781a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 -- 172.21.15.123:0/1642024779 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41781a5a30 con 0x7f4178076060 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177400640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.018+0000 7f4177e00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9f0 tx=0x7f416800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.019+0000 7f4174c00640 1 -- 172.21.15.123:0/1642024779 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416800ccc0 con 0x7f4178076060 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.019+0000 7f4174c00640 1 -- 172.21.15.123:0/1642024779 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4168010910 con 0x7f4178076060 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.019+0000 7f4174c00640 1 -- 172.21.15.123:0/1642024779 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168018960 con 0x7f4178076060 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.019+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41781a5d20 con 0x7f4178076060 2024-03-24T00:47:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.019+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41781a6200 con 0x7f4178076060 2024-03-24T00:47:49.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4174c00640 1 -- 172.21.15.123:0/1642024779 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4168010ac0 con 0x7f4178076060 2024-03-24T00:47:49.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 Processor -- start 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 -- start start 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b61470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b66540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0ac6250 con 0x55d8d0b61050 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0a6ea30 con 0x55d8d0b61a40 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.023+0000 7f4181e34740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0ac22f0 con 0x55d8d0b66bb0 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b66540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37342/0 (socket says 172.21.15.123:37342) 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 -- 172.21.15.123:0/2321370534 learned_addr learned my addr 172.21.15.123:0/2321370534 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f417d200640 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b61470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 -- 172.21.15.123:0/2321370534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 msgr2=0x55d8d0b66540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b66540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 -- 172.21.15.123:0/2321370534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 msgr2=0x55d8d0b61470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b61470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177400640 1 -- 172.21.15.123:0/2321370534 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8d0a59690 con 0x55d8d0b66bb0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4177e00640 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b66540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.024+0000 7f4174c00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 0x7f415c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f4174c00640 1 -- 172.21.15.123:0/1642024779 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41680ceda0 con 0x7f4178076060 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f417d200640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 0x7f415c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f4177400640 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f41640115f0 tx=0x7f4164011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7636fbf24ed3ad1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f416d600640 1 -- 172.21.15.123:0/2321370534 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416401e070 con 0x55d8d0b66bb0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f416d600640 1 -- 172.21.15.123:0/2321370534 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f416400fba0 con 0x55d8d0b66bb0 2024-03-24T00:47:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.025+0000 7f416d600640 1 -- 172.21.15.123:0/2321370534 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4164019d80 con 0x55d8d0b66bb0 2024-03-24T00:47:49.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 -- 172.21.15.123:0/2321370534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 msgr2=0x55d8d0b69040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f41640115f0 tx=0x7f4164011ab0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f417d200640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 0x7f415c0a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f417002c640 tx=0x7f4170032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 -- 172.21.15.123:0/2321370534 shutdown_connections 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b69040 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b66540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 --2- 172.21.15.123:0/2321370534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b61470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.026+0000 7f4181e34740 1 -- 172.21.15.123:0/2321370534 >> 172.21.15.123:0/2321370534 conn(0x55d8d0b29200 msgr2=0x55d8d0b5c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.027+0000 7f4181e34740 1 -- 172.21.15.123:0/2321370534 shutdown_connections 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.027+0000 7f4181e34740 1 -- 172.21.15.123:0/2321370534 wait complete. 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.027+0000 7f4181e34740 1 -- start start 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.027+0000 7f4181e34740 1 Processor -- start 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 0x55d8d0b6b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b72590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0b69b60 con 0x55d8d0b61a40 2024-03-24T00:47:49.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0a81e80 con 0x55d8d0b66bb0 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8d0b69ce0 con 0x55d8d0b61050 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4181e34740 10 client.0 _ll_register_callbacks cb 0x55d8d0a94130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4177e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f417d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b72590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4177e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37348/0 (socket says 172.21.15.123:37348) 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4177e00640 1 -- 172.21.15.123:0/22141353 learned_addr learned my addr 172.21.15.123:0/22141353 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f417d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b72590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50002/0 (socket says 172.21.15.123:50002) 2024-03-24T00:47:49.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.028+0000 7f4177400640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 0x55d8d0b6b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 -- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 msgr2=0x55d8d0b72590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b72590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 -- 172.21.15.123:0/22141353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 msgr2=0x55d8d0b6b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 0x55d8d0b6b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 -- 172.21.15.123:0/22141353 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8d08a47d0 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177400640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 0x55d8d0b6b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4177e00640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9c0 tx=0x7f4168097b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd272c3058f2fa72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4162000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416809b100 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f416400fba0 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41680a3a00 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.029+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f416809b7a0 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.030+0000 7f4181e34740 10 client.26575 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.030+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d8d07da3f0 con 0x55d8d0b61050 2024-03-24T00:47:49.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.030+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f416400fba0 con 0x55d8d0b61050 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4168018b00 con 0x55d8d0b61050 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 10 client.26575 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 10 client.26575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 20 client.26575 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0b76db0 0x55d8d0b79200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 4 client.26575 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8d0ad3790 con 0x55d8d0b76db0 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.031+0000 7f4177400640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0b76db0 0x55d8d0b79200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.032+0000 7f4177400640 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0b76db0 0x55d8d0b79200 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62e835c8eaa6d327 server_cookie=56e545a2f0e9e9d9 in_seq=0 out_seq=0 2024-03-24T00:47:49.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.032+0000 7f4162000640 10 client.26575 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:49.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.033+0000 7f4162000640 1 -- 172.21.15.123:0/22141353 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d8d0ad3790 con 0x55d8d0b76db0 2024-03-24T00:47:49.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.033+0000 7f4162000640 10 client.26575 handle_command_reply: tid=0 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.033+0000 7f4181e34740 1 client.26575 shutdown 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.033+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 msgr2=0x55d8d0b6b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.033+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9c0 tx=0x7f4168097b00 comp rx=0 tx=0).stop 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 shutdown_connections 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b72590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61a40 0x55d8d0b6b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0b76db0 0x55d8d0b79200 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 --2- 172.21.15.123:0/22141353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6b080 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 >> 172.21.15.123:0/22141353 conn(0x55d8d0b29200 msgr2=0x55d8d0b5bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4162000640 0 client.26575 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 shutdown_connections 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.034+0000 7f4181e34740 1 -- 172.21.15.123:0/22141353 wait complete. 2024-03-24T00:47:49.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.035+0000 7f4181e34740 20 client.26575 trim_cache size 0 max 16384 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 Processor -- start 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 -- start start 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b60490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 0x55d8d0b72550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781acf60 con 0x55d8d0b61050 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781af520 con 0x55d8d0b66bb0 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.056+0000 7f4181e34740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781ad560 con 0x55d8d0b61a40 2024-03-24T00:47:49.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f4177400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 0x55d8d0b72550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f4177400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 0x55d8d0b72550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:37354/0 (socket says 172.21.15.123:37354) 2024-03-24T00:47:49.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f4177400640 1 -- 172.21.15.123:0/1361004349 learned_addr learned my addr 172.21.15.123:0/1361004349 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:49.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f4177e00640 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b60490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 -- 172.21.15.123:0/1361004349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 msgr2=0x55d8d0b72550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 0x55d8d0b72550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 -- 172.21.15.123:0/1361004349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 msgr2=0x55d8d0b60490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b60490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f417d200640 1 -- 172.21.15.123:0/1361004349 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8d079a0f0 con 0x55d8d0b66bb0 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.057+0000 7f4177e00640 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b60490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.058+0000 7f417d200640 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b6b3a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4170037040 tx=0x7f4170041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5c683c260d2097e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.058+0000 7f416cc00640 1 -- 172.21.15.123:0/1361004349 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170039040 con 0x55d8d0b66bb0 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.058+0000 7f416cc00640 1 -- 172.21.15.123:0/1361004349 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4170004040 con 0x55d8d0b66bb0 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.058+0000 7f416cc00640 1 -- 172.21.15.123:0/1361004349 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417004e6c0 con 0x55d8d0b66bb0 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 -- 172.21.15.123:0/1361004349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 msgr2=0x55d8d0b6b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b6b3a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4170037040 tx=0x7f4170041000 comp rx=0 tx=0).stop 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 -- 172.21.15.123:0/1361004349 shutdown_connections 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b6b3a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61a40 0x55d8d0b72550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 --2- 172.21.15.123:0/1361004349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b61050 0x55d8d0b60490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 -- 172.21.15.123:0/1361004349 >> 172.21.15.123:0/1361004349 conn(0x55d8d0b29200 msgr2=0x55d8d0b6ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.059+0000 7f4181e34740 1 -- 172.21.15.123:0/1361004349 shutdown_connections 2024-03-24T00:47:49.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.060+0000 7f4181e34740 1 -- 172.21.15.123:0/1361004349 wait complete. 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.060+0000 7f4181e34740 1 -- start start 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.060+0000 7f4181e34740 1 Processor -- start 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781adba0 con 0x55d8d0b66bb0 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781aeec0 con 0x55d8d0b61a40 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781ae200 con 0x55d8d0b61050 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4181e34740 10 client.0 _ll_register_callbacks cb 0x7f41680d24d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:49.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:50034/0 (socket says 172.21.15.123:50034) 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 -- 172.21.15.123:0/2841825042 learned_addr learned my addr 172.21.15.123:0/2841825042 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f417d200640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177e00640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 -- 172.21.15.123:0/2841825042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 msgr2=0x55d8d0b6fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 -- 172.21.15.123:0/2841825042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b66bb0 msgr2=0x55d8d0b711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.061+0000 7f4177400640 1 -- 172.21.15.123:0/2841825042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8d0b71790 con 0x55d8d0b61a40 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.062+0000 7f4177400640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f41640098e0 tx=0x7f4164009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8987469c7a3ed75a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.062+0000 7f4162000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.062+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416402b070 con 0x55d8d0b61a40 2024-03-24T00:47:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.062+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41640270f0 con 0x55d8d0b61a40 2024-03-24T00:47:49.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.063+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4170004040 con 0x55d8d0b61a40 2024-03-24T00:47:49.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.063+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4164037050 con 0x55d8d0b61a40 2024-03-24T00:47:49.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.063+0000 7f4181e34740 10 client.26570 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:49.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.063+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d8d0bd2690 con 0x55d8d0b61a40 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.063+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4170004040 con 0x55d8d0b61a40 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4164005480 con 0x55d8d0b61a40 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 10 client.26570 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 10 client.26570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 20 client.26570 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0bd2e30 0x55d8d0bd5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 4 client.26570 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d8d0acdae0 con 0x55d8d0bd2e30 2024-03-24T00:47:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.064+0000 7f4177e00640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0bd2e30 0x55d8d0bd5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4177e00640 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0bd2e30 0x55d8d0bd5250 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd2aaf85062ba144 server_cookie=5dbe8a3719cfc14d in_seq=0 out_seq=0 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4162000640 10 client.26570 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4162000640 1 -- 172.21.15.123:0/2841825042 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55d8d0acdae0 con 0x55d8d0bd2e30 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4162000640 10 client.26570 handle_command_reply: tid=0 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 client.26570 shutdown 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 msgr2=0x55d8d0b70190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f41640098e0 tx=0x7f4164009370 comp rx=0 tx=0).stop 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 shutdown_connections 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8d0b66bb0 0x55d8d0b711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8d0b61a40 0x55d8d0b70190 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d8d0bd2e30 0x55d8d0bd5250 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 --2- 172.21.15.123:0/2841825042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8d0b61050 0x55d8d0b6fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 >> 172.21.15.123:0/2841825042 conn(0x55d8d0b29200 msgr2=0x55d8d0b62440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.065+0000 7f4162000640 0 client.26570 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.066+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 shutdown_connections 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.066+0000 7f4181e34740 1 -- 172.21.15.123:0/2841825042 wait complete. 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.066+0000 7f4181e34740 20 client.26570 trim_cache size 0 max 16384 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (3 inodes in the stack)", 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "77aff907-ed5f-41b8-92a8-b4b53aaa4a28": { 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:47:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "77aff907-ed5f-41b8-92a8-b4b53aaa4a28", 2024-03-24T00:47:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:47:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:47:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:47:49.071 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:47:49.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.066+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 msgr2=0x7f415c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.066+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 0x7f415c0a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f417002c640 tx=0x7f4170032590 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 msgr2=0x7f417819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9f0 tx=0x7f416800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 shutdown_connections 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f415c0a2880 0x7f415c0a4d40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4178077c40 0x7f41781a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41780772a0 0x7f417819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 --2- 172.21.15.123:0/1642024779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4178076060 0x7f417819de10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 >> 172.21.15.123:0/1642024779 conn(0x7f41780fed10 msgr2=0x7f4178100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 shutdown_connections 2024-03-24T00:47:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:47:49.067+0000 7f417dc00640 1 -- 172.21.15.123:0/1642024779 wait complete. 2024-03-24T00:47:49.086 INFO:tasks.cephfs.filesystem:scrub status for tag:77aff907-ed5f-41b8-92a8-b4b53aaa4a28 - {'path': '/', 'tag': '77aff907-ed5f-41b8-92a8-b4b53aaa4a28', 'options': 'recursive,force'} 2024-03-24T00:47:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:49 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:47:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:49 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:47:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:49 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:47:49.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:49 smithi204 ceph-mon[36345]: pgmap v683: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 25 MiB/s wr, 4.55k op/s 2024-03-24T00:47:49.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:49 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:47:49.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:49 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:47:49.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:49 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:47:49.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:49 smithi123 ceph-mon[31487]: pgmap v683: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 25 MiB/s wr, 4.55k op/s 2024-03-24T00:47:49.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:49 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:47:49.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:49 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:47:49.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:49 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:47:49.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:49 smithi177 ceph-mon[36034]: pgmap v683: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 25 MiB/s wr, 4.55k op/s 2024-03-24T00:47:52.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:52 smithi177 ceph-mon[36034]: pgmap v684: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 27 MiB/s wr, 4.87k op/s 2024-03-24T00:47:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:52 smithi204 ceph-mon[36345]: pgmap v684: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 27 MiB/s wr, 4.87k op/s 2024-03-24T00:47:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:52 smithi123 ceph-mon[31487]: pgmap v684: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 27 MiB/s wr, 4.87k op/s 2024-03-24T00:47:54.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:54 smithi177 ceph-mon[36034]: pgmap v685: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 4.94k op/s 2024-03-24T00:47:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:54 smithi204 ceph-mon[36345]: pgmap v685: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 4.94k op/s 2024-03-24T00:47:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:54 smithi123 ceph-mon[31487]: pgmap v685: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 4.94k op/s 2024-03-24T00:47:56.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:56 smithi177 ceph-mon[36034]: pgmap v686: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 27 MiB/s wr, 4.91k op/s 2024-03-24T00:47:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:56 smithi204 ceph-mon[36345]: pgmap v686: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 27 MiB/s wr, 4.91k op/s 2024-03-24T00:47:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:56 smithi123 ceph-mon[31487]: pgmap v686: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 27 MiB/s wr, 4.91k op/s 2024-03-24T00:47:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:58 smithi177 ceph-mon[36034]: pgmap v687: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 5.13k op/s 2024-03-24T00:47:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:58 smithi204 ceph-mon[36345]: pgmap v687: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 5.13k op/s 2024-03-24T00:47:58.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:58 smithi123 ceph-mon[31487]: pgmap v687: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 5.13k op/s 2024-03-24T00:47:59.014 INFO:tasks.workunit.client.0.smithi123.stdout:Syncing()...131 sec 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:FFSB benchmark finished at: Sun Mar 24 00:47:59 2024 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:Results: 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:Benchmark took 441.47 sec 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:Total Results 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:=============== 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: ======= ============ ========= ======= =========== ========== 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: write : 7403520 16770.09 100.000% 100.000% 65.5MB/sec 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:- 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:16770.09 Transactions per Second 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.015 INFO:tasks.workunit.client.0.smithi123.stdout:Throughput Results 2024-03-24T00:47:59.016 INFO:tasks.workunit.client.0.smithi123.stdout:=================== 2024-03-24T00:47:59.016 INFO:tasks.workunit.client.0.smithi123.stdout:Write Throughput: 65.5MB/sec 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout:System Call Latency statistics in millisecs 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout:===== 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: Min Avg Max Total Calls 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: ======== ======== ======== ============ 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout:[ open] 0.019000 0.240662 251.401001 5784 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-24T00:47:59.089 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[4] 200.000000 - 500.000000 : 1 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[7] 20.000000 - 50.000000 : 2 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[8] 10.000000 - 20.000000 : 19 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[9] 5.000000 - 10.000000 : 43 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[10] 2.000000 - 5.000000 : 57 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[11] 1.000000 - 2.000000 : 13 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[12] 0.500000 - 1.000000 : 19 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[13] 0.200000 - 0.500000 : 21 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[14] 0.100000 - 0.200000 : 374 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[15] 0.050000 - 0.100000 : 1978 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[16] 0.020000 - 0.050000 : 3255 2024-03-24T00:47:59.090 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[17] 0.010000 - 0.020000 : 2 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout:[ write] 0.003000 1.304877 10538.264648 7403520 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[0] 5000.000000 - 10000.000000 : 94 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[1] 2000.000000 - 5000.000000 : 303 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[2] 1000.000000 - 2000.000000 : 291 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[3] 500.000000 - 1000.000000 : 254 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[4] 200.000000 - 500.000000 : 3367 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[5] 100.000000 - 200.000000 : 10655 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[6] 50.000000 - 100.000000 : 59600 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[7] 20.000000 - 50.000000 : 22542 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[8] 10.000000 - 20.000000 : 1989 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[9] 5.000000 - 10.000000 : 1602 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[10] 2.000000 - 5.000000 : 1883 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[11] 1.000000 - 2.000000 : 1167 2024-03-24T00:47:59.091 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[12] 0.500000 - 1.000000 : 1545 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[13] 0.200000 - 0.500000 : 4058 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[14] 0.100000 - 0.200000 : 11117 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[15] 0.050000 - 0.100000 : 60784 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[16] 0.020000 - 0.050000 : 903094 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[17] 0.010000 - 0.020000 : 2461426 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[18] 0.000000 - 0.010000 : 3857746 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout:[ lseek] 0.000000 0.001942 64.045998 7403520 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-03-24T00:47:59.092 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[7] 20.000000 - 50.000000 : 22 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[8] 10.000000 - 20.000000 : 138 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[9] 5.000000 - 10.000000 : 114 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[10] 2.000000 - 5.000000 : 142 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[11] 1.000000 - 2.000000 : 110 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[12] 0.500000 - 1.000000 : 113 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[13] 0.200000 - 0.500000 : 252 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[14] 0.100000 - 0.200000 : 435 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[15] 0.050000 - 0.100000 : 1279 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[16] 0.020000 - 0.050000 : 5858 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[17] 0.010000 - 0.020000 : 6725 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[18] 0.000000 - 0.010000 : 7388330 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout:[ close] 0.004000 0.019469 22.066000 5784 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-24T00:47:59.093 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[7] 20.000000 - 50.000000 : 1 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[13] 0.200000 - 0.500000 : 5 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[14] 0.100000 - 0.200000 : 8 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[15] 0.050000 - 0.100000 : 21 2024-03-24T00:47:59.094 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[16] 0.020000 - 0.050000 : 461 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[17] 0.010000 - 0.020000 : 3124 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: msec_range[18] 0.000000 - 0.010000 : 2162 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout:1.3% User Time 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout:23.2% System Time 2024-03-24T00:47:59.095 INFO:tasks.workunit.client.0.smithi123.stdout:24.5% CPU Utilization 2024-03-24T00:47:59.105 INFO:tasks.workunit.client.0.smithi123.stderr:+ cd .. 2024-03-24T00:47:59.106 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -r tmp ffsb 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 -- 172.21.15.123:0/3388067444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4540756e0 msgr2=0x7ff454073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 --2- 172.21.15.123:0/3388067444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4540756e0 0x7ff454073b30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff444009930 tx=0x7ff44402bc20 comp rx=0 tx=0).stop 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 -- 172.21.15.123:0/3388067444 shutdown_connections 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 --2- 172.21.15.123:0/3388067444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 0x7ff45410fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 --2- 172.21.15.123:0/3388067444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff454074100 0x7ff454074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.778+0000 7ff458e00640 1 --2- 172.21.15.123:0/3388067444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4540756e0 0x7ff454073b30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.779+0000 7ff458e00640 1 -- 172.21.15.123:0/3388067444 >> 172.21.15.123:0/3388067444 conn(0x7ff4540fca80 msgr2=0x7ff4540feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.779+0000 7ff458e00640 1 -- 172.21.15.123:0/3388067444 shutdown_connections 2024-03-24T00:47:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.779+0000 7ff458e00640 1 -- 172.21.15.123:0/3388067444 wait complete. 2024-03-24T00:47:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 Processor -- start 2024-03-24T00:47:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 -- start start 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4540756e0 0x7ff45419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 0x7ff4541a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541137d0 con 0x7ff45410dae0 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454113650 con 0x7ff454074100 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.780+0000 7ff458e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454113950 con 0x7ff4540756e0 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:57462/0 (socket says 172.21.15.123:57462) 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 -- 172.21.15.123:0/1960185499 learned_addr learned my addr 172.21.15.123:0/1960185499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff452a00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4540756e0 0x7ff45419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 0x7ff4541a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 -- 172.21.15.123:0/1960185499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4540756e0 msgr2=0x7ff45419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4540756e0 0x7ff45419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 -- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 msgr2=0x7ff4541a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 0x7ff4541a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 -- 172.21.15.123:0/1960185499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4541a5a60 con 0x7ff454074100 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.781+0000 7ff453400640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff444000c00 tx=0x7ff44402c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.782+0000 7ff44be00640 1 -- 172.21.15.123:0/1960185499 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444043020 con 0x7ff454074100 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.782+0000 7ff44be00640 1 -- 172.21.15.123:0/1960185499 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4440054e0 con 0x7ff454074100 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.782+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4541a5cf0 con 0x7ff454074100 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.782+0000 7ff44be00640 1 -- 172.21.15.123:0/1960185499 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444003c10 con 0x7ff454074100 2024-03-24T00:47:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.782+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4541a6190 con 0x7ff454074100 2024-03-24T00:47:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.785+0000 7ff44be00640 1 -- 172.21.15.123:0/1960185499 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff444046020 con 0x7ff454074100 2024-03-24T00:47:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.785+0000 7ff45d1cb740 1 Processor -- start 2024-03-24T00:47:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 -- start start 2024-03-24T00:47:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e278e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e2a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2de53b0 con 0x55f0e2e22de0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2d87830 con 0x55f0e2e223f0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2d305d0 con 0x55f0e2e27f50 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff453e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff453e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:57466/0 (socket says 172.21.15.123:57466) 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff453e00640 1 -- 172.21.15.123:0/4099514557 learned_addr learned my addr 172.21.15.123:0/4099514557 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.786+0000 7ff453400640 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e278e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 -- 172.21.15.123:0/4099514557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 msgr2=0x55f0e2e2a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e2a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 -- 172.21.15.123:0/4099514557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 msgr2=0x55f0e2e278e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e278e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 -- 172.21.15.123:0/4099514557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0e2d482c0 con 0x55f0e2e223f0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff453e00640 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009fd0 tx=0x7ff44c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dfffadb495e8f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff44be00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 0x7ff4380a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff448c00640 1 -- 172.21.15.123:0/4099514557 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c011700 con 0x55f0e2e223f0 2024-03-24T00:47:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff448c00640 1 -- 172.21.15.123:0/4099514557 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44c011d80 con 0x55f0e2e223f0 2024-03-24T00:47:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.787+0000 7ff448c00640 1 -- 172.21.15.123:0/4099514557 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c01a380 con 0x55f0e2e223f0 2024-03-24T00:47:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff44be00640 1 -- 172.21.15.123:0/1960185499 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff4440f0c30 con 0x7ff454074100 2024-03-24T00:47:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff453400640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 0x7ff4380a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 -- 172.21.15.123:0/4099514557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 msgr2=0x55f0e2e22810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009fd0 tx=0x7ff44c010860 comp rx=0 tx=0).stop 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 -- 172.21.15.123:0/4099514557 shutdown_connections 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e2a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e278e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/4099514557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e22810 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.788+0000 7ff45d1cb740 1 -- 172.21.15.123:0/4099514557 >> 172.21.15.123:0/4099514557 conn(0x55f0e2e1d960 msgr2=0x55f0e2e1dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.789+0000 7ff453400640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 0x7ff4380a4cb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4440b9a70 tx=0x7ff4440f4670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.789+0000 7ff45d1cb740 1 -- 172.21.15.123:0/4099514557 shutdown_connections 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.789+0000 7ff45d1cb740 1 -- 172.21.15.123:0/4099514557 wait complete. 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.789+0000 7ff45d1cb740 1 -- start start 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.789+0000 7ff45d1cb740 1 Processor -- start 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e2c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 0x55f0e2e33b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2d43a20 con 0x55f0e2e27f50 2024-03-24T00:47:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2d838d0 con 0x55f0e2e223f0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0e2e2af00 con 0x55f0e2e22de0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff45d1cb740 10 client.0 _ll_register_callbacks cb 0x55f0e2d801c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33134/0 (socket says 172.21.15.123:33134) 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 -- 172.21.15.123:0/3570819970 learned_addr learned my addr 172.21.15.123:0/3570819970 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff453e00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e2c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff453e00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 0x55f0e2e33b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 -- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 msgr2=0x55f0e2e2c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e2c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 -- 172.21.15.123:0/3570819970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 msgr2=0x55f0e2e33b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 0x55f0e2e33b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.790+0000 7ff452a00640 1 -- 172.21.15.123:0/3570819970 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0e2a9c210 con 0x55f0e2e22de0 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff453e00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 0x55f0e2e33b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff452a00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff4400116f0 tx=0x7ff440011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e3ffb469c2f976 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff43d600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44001e060 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff44000fd10 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff44c011d80 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.791+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff440019430 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 10 client.26590 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f0e2b68620 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44c011d80 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff440011ce0 con 0x55f0e2e22de0 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 10 client.26590 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 10 client.26590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 20 client.26590 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e38540 0x55f0e2e3a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 4 client.26590 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0e2d94d70 con 0x55f0e2e38540 2024-03-24T00:47:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.792+0000 7ff453e00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e38540 0x55f0e2e3a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.793+0000 7ff453e00640 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e38540 0x55f0e2e3a960 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce6ec5125af7a5d server_cookie=a151b964f380b736 in_seq=0 out_seq=0 2024-03-24T00:47:59.796 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.793+0000 7ff43d600640 10 client.26590 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.794+0000 7ff43d600640 1 -- 172.21.15.123:0/3570819970 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f0e2d94d70 con 0x55f0e2e38540 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.794+0000 7ff43d600640 10 client.26590 handle_command_reply: tid=0 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 client.26590 shutdown 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 msgr2=0x55f0e2e2cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff4400116f0 tx=0x7ff440011bb0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 shutdown_connections 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e27f50 0x55f0e2e33b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e38540 0x55f0e2e3a960 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e2cb20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3570819970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e2c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff43d600640 0 client.26590 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 >> 172.21.15.123:0/3570819970 conn(0x55f0e2e1d960 msgr2=0x55f0e2e1cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 shutdown_connections 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3570819970 wait complete. 2024-03-24T00:47:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.795+0000 7ff45d1cb740 20 client.26590 trim_cache size 0 max 16384 2024-03-24T00:47:59.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:47:59 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 Processor -- start 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 -- start start 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e1c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e1c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541aced0 con 0x55f0e2e22de0 2024-03-24T00:47:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541af490 con 0x55f0e2e223f0 2024-03-24T00:47:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.824+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541ad4d0 con 0x55f0e2e27f50 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff453e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e1c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44998/0 (socket says 172.21.15.123:44998) 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 -- 172.21.15.123:0/1857670036 learned_addr learned my addr 172.21.15.123:0/1857670036 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff453e00640 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e1c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 -- 172.21.15.123:0/1857670036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 msgr2=0x55f0e2e1c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e1c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 -- 172.21.15.123:0/1857670036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 msgr2=0x55f0e2e1c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e1c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 -- 172.21.15.123:0/1857670036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0e2e3afc0 con 0x55f0e2e22de0 2024-03-24T00:47:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.825+0000 7ff452a00640 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff440000c00 tx=0x7ff440012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b2ff742076da6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.826+0000 7ff43fe00640 1 -- 172.21.15.123:0/1857670036 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440029070 con 0x55f0e2e22de0 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.826+0000 7ff43fe00640 1 -- 172.21.15.123:0/1857670036 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff440005520 con 0x55f0e2e22de0 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.826+0000 7ff43fe00640 1 -- 172.21.15.123:0/1857670036 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000ed40 con 0x55f0e2e22de0 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 -- 172.21.15.123:0/1857670036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 msgr2=0x55f0e2e33b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff440000c00 tx=0x7ff440012f00 comp rx=0 tx=0).stop 2024-03-24T00:47:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 -- 172.21.15.123:0/1857670036 shutdown_connections 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e27f50 0x55f0e2e1c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e22de0 0x55f0e2e33b20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/1857670036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e223f0 0x55f0e2e1c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 -- 172.21.15.123:0/1857670036 >> 172.21.15.123:0/1857670036 conn(0x55f0e2e1d960 msgr2=0x55f0e2e24140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.827+0000 7ff45d1cb740 1 -- 172.21.15.123:0/1857670036 shutdown_connections 2024-03-24T00:47:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.828+0000 7ff45d1cb740 1 -- 172.21.15.123:0/1857670036 wait complete. 2024-03-24T00:47:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.828+0000 7ff45d1cb740 1 -- start start 2024-03-24T00:47:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.828+0000 7ff45d1cb740 1 Processor -- start 2024-03-24T00:47:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e223f0 0x55f0e2e31220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e27f50 0x55f0e2e326b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541adb10 con 0x55f0e2e223f0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541aee30 con 0x55f0e2e27f50 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541ae170 con 0x55f0e2e22de0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff45d1cb740 10 client.0 _ll_register_callbacks cb 0x7ff440009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff453e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e223f0 0x55f0e2e31220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff452a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff452a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33156/0 (socket says 172.21.15.123:33156) 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff452a00640 1 -- 172.21.15.123:0/3280894930 learned_addr learned my addr 172.21.15.123:0/3280894930 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.829+0000 7ff453e00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e27f50 0x55f0e2e326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 -- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e27f50 msgr2=0x55f0e2e326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e27f50 0x55f0e2e326b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 -- 172.21.15.123:0/3280894930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e223f0 msgr2=0x55f0e2e31220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e223f0 0x55f0e2e31220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 -- 172.21.15.123:0/3280894930 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0e2e32ca0 con 0x55f0e2e22de0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff452a00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c90 tx=0x7ff440005300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99b838f7b469c8e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff43d600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44003a060 con 0x55f0e2e22de0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff44000ea80 con 0x55f0e2e22de0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.830+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff440005520 con 0x55f0e2e22de0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff440029030 con 0x55f0e2e22de0 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 10 client.26596 fetch_fsmap learned FSMap version 19 2024-03-24T00:47:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f0e2e93db0 con 0x55f0e2e22de0 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff440005520 con 0x55f0e2e22de0 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff440011ce0 con 0x55f0e2e22de0 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 10 client.26596 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 10 client.26596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 20 client.26596 populate_metadata read hostname 'smithi123' 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e94580 0x55f0e2e969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 4 client.26596 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.831+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f0e2d8f0c0 con 0x55f0e2e94580 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.832+0000 7ff453e00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e94580 0x55f0e2e969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.832+0000 7ff43d600640 10 client.26596 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.832+0000 7ff453e00640 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e94580 0x55f0e2e969a0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7505e580d8271a92 server_cookie=abfc6d9b6310ff0e in_seq=0 out_seq=0 2024-03-24T00:47:59.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:47:59 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff43d600640 1 -- 172.21.15.123:0/3280894930 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f0e2d8f0c0 con 0x55f0e2e94580 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff43d600640 10 client.26596 handle_command_reply: tid=0 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 client.26596 shutdown 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 msgr2=0x55f0e2e31760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c90 tx=0x7ff440005300 comp rx=0 tx=0).stop 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 shutdown_connections 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0e2e27f50 0x55f0e2e326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55f0e2e94580 0x55f0e2e969a0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0e2e22de0 0x55f0e2e31760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 --2- 172.21.15.123:0/3280894930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0e2e223f0 0x55f0e2e31220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff43d600640 0 client.26596 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:47:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.881+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 >> 172.21.15.123:0/3280894930 conn(0x55f0e2e1d960 msgr2=0x55f0e2e20bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 shutdown_connections 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff45d1cb740 1 -- 172.21.15.123:0/3280894930 wait complete. 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff45d1cb740 20 client.26596 trim_cache size 0 max 16384 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:47:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 msgr2=0x7ff4380a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 0x7ff4380a4cb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4440b9a70 tx=0x7ff4440f4670 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 msgr2=0x7ff45419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.882+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff444000c00 tx=0x7ff44402c630 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 shutdown_connections 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff4380a27f0 0x7ff4380a4cb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45410dae0 0x7ff4541a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4540756e0 0x7ff45419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 --2- 172.21.15.123:0/1960185499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff454074100 0x7ff45419de40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 >> 172.21.15.123:0/1960185499 conn(0x7ff4540fca80 msgr2=0x7ff454100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 shutdown_connections 2024-03-24T00:47:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:47:59.883+0000 7ff458e00640 1 -- 172.21.15.123:0/1960185499 wait complete. 2024-03-24T00:47:59.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:47:59 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:00 smithi204 ceph-mon[36345]: pgmap v688: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 22 MiB/s wr, 4.35k op/s 2024-03-24T00:48:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:00 smithi123 ceph-mon[31487]: pgmap v688: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 22 MiB/s wr, 4.35k op/s 2024-03-24T00:48:00.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:00 smithi177 ceph-mon[36034]: pgmap v688: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 22 MiB/s wr, 4.35k op/s 2024-03-24T00:48:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:02 smithi204 ceph-mon[36345]: pgmap v689: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 24 MiB/s wr, 4.71k op/s 2024-03-24T00:48:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:02 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:48:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:02 smithi123 ceph-mon[31487]: pgmap v689: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 24 MiB/s wr, 4.71k op/s 2024-03-24T00:48:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:02 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:48:02.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:02 smithi177 ceph-mon[36034]: pgmap v689: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 24 MiB/s wr, 4.71k op/s 2024-03-24T00:48:02.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:02 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:48:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:04 smithi204 ceph-mon[36345]: pgmap v690: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 3.40k op/s 2024-03-24T00:48:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:04 smithi123 ceph-mon[31487]: pgmap v690: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 3.40k op/s 2024-03-24T00:48:04.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:04 smithi177 ceph-mon[36034]: pgmap v690: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 3.40k op/s 2024-03-24T00:48:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:06 smithi204 ceph-mon[36345]: pgmap v691: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 2.64k op/s 2024-03-24T00:48:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:06 smithi123 ceph-mon[31487]: pgmap v691: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 2.64k op/s 2024-03-24T00:48:06.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:06 smithi177 ceph-mon[36034]: pgmap v691: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 2.64k op/s 2024-03-24T00:48:08.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:08 smithi177 ceph-mon[36034]: pgmap v692: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.7 MiB/s wr, 2.09k op/s 2024-03-24T00:48:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:08 smithi204 ceph-mon[36345]: pgmap v692: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.7 MiB/s wr, 2.09k op/s 2024-03-24T00:48:08.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:08 smithi123 ceph-mon[31487]: pgmap v692: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.7 MiB/s wr, 2.09k op/s 2024-03-24T00:48:09.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:09 smithi204 ceph-mon[36345]: pgmap v693: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.8 MiB/s wr, 667 op/s 2024-03-24T00:48:09.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:09 smithi123 ceph-mon[31487]: pgmap v693: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.8 MiB/s wr, 667 op/s 2024-03-24T00:48:09.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:09 smithi177 ceph-mon[36034]: pgmap v693: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.8 MiB/s wr, 667 op/s 2024-03-24T00:48:12.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:12 smithi177 ceph-mon[36034]: pgmap v694: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.8 MiB/s wr, 660 op/s 2024-03-24T00:48:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:12 smithi204 ceph-mon[36345]: pgmap v694: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.8 MiB/s wr, 660 op/s 2024-03-24T00:48:12.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:12 smithi123 ceph-mon[31487]: pgmap v694: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.8 MiB/s wr, 660 op/s 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 -- 172.21.15.123:0/930113254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd401054b0 msgr2=0x7fdd4010bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 --2- 172.21.15.123:0/930113254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd401054b0 0x7fdd4010bd40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009ca0 tx=0x7fdd3802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 -- 172.21.15.123:0/930113254 shutdown_connections 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 --2- 172.21.15.123:0/930113254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd401054b0 0x7fdd4010bd40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 --2- 172.21.15.123:0/930113254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd40104af0 0x7fdd40104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 --2- 172.21.15.123:0/930113254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40103900 0x7fdd40103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 -- 172.21.15.123:0/930113254 >> 172.21.15.123:0/930113254 conn(0x7fdd400fee90 msgr2=0x7fdd401012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 -- 172.21.15.123:0/930113254 shutdown_connections 2024-03-24T00:48:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.017+0000 7fdd46600640 1 -- 172.21.15.123:0/930113254 wait complete. 2024-03-24T00:48:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.018+0000 7fdd46600640 1 Processor -- start 2024-03-24T00:48:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 -- start start 2024-03-24T00:48:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40104af0 0x7fdd4019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 0x7fdd401a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4010f580 con 0x7fdd40104af0 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4010f400 con 0x7fdd40103900 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd46600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4010f700 con 0x7fdd401054b0 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33970/0 (socket says 172.21.15.123:33970) 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd3f400640 1 -- 172.21.15.123:0/833365964 learned_addr learned my addr 172.21.15.123:0/833365964 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40104af0 0x7fdd4019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.019+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 0x7fdd401a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 -- 172.21.15.123:0/833365964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 msgr2=0x7fdd401a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 0x7fdd401a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 -- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40104af0 msgr2=0x7fdd4019a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40104af0 0x7fdd4019a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 -- 172.21.15.123:0/833365964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd401a1730 con 0x7fdd40103900 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 0x7fdd401a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd3f400640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd37e00640 1 -- 172.21.15.123:0/833365964 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3000cd80 con 0x7fdd40103900 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd37e00640 1 -- 172.21.15.123:0/833365964 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd300035f0 con 0x7fdd40103900 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd37e00640 1 -- 172.21.15.123:0/833365964 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30010690 con 0x7fdd40103900 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd401a1a20 con 0x7fdd40103900 2024-03-24T00:48:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.020+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd401a1f20 con 0x7fdd40103900 2024-03-24T00:48:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.023+0000 7fdd37e00640 1 -- 172.21.15.123:0/833365964 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdd300037a0 con 0x7fdd40103900 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 Processor -- start 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 -- start start 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d61600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d666d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d66d40 0x558277d691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277d243b0 con 0x558277d66d40 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277cc6830 con 0x558277d61bd0 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd4940d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277c6f5d0 con 0x558277d611e0 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d61600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d666d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33978/0 (socket says 172.21.15.123:33978) 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd3f400640 1 -- 172.21.15.123:0/2451865739 learned_addr learned my addr 172.21.15.123:0/2451865739 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.024+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d66d40 0x558277d691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 -- 172.21.15.123:0/2451865739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 msgr2=0x558277d666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d666d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 -- 172.21.15.123:0/2451865739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d66d40 msgr2=0x558277d691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d66d40 0x558277d691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 -- 172.21.15.123:0/2451865739 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558277c872c0 con 0x558277d611e0 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd37e00640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 0x7fdd240a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d61600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd38037040 tx=0x7fdd38005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7ad01a59fdcf3ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd37e00640 1 -- 172.21.15.123:0/833365964 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdd300cee60 con 0x7fdd40103900 2024-03-24T00:48:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd3f400640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 0x7fdd240a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd34c00640 1 -- 172.21.15.123:0/2451865739 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38043070 con 0x558277d611e0 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd34c00640 1 -- 172.21.15.123:0/2451865739 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd38032c90 con 0x558277d611e0 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.025+0000 7fdd34c00640 1 -- 172.21.15.123:0/2451865739 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38039040 con 0x558277d611e0 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 -- 172.21.15.123:0/2451865739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 msgr2=0x558277d61600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d61600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd38037040 tx=0x7fdd38005ef0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd3f400640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 0x7fdd240a4d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdd300cf100 tx=0x7fdd300d28a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 -- 172.21.15.123:0/2451865739 shutdown_connections 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d66d40 0x558277d691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 --2- 172.21.15.123:0/2451865739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d61600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 -- 172.21.15.123:0/2451865739 >> 172.21.15.123:0/2451865739 conn(0x558277d5c750 msgr2=0x558277d5cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 -- 172.21.15.123:0/2451865739 shutdown_connections 2024-03-24T00:48:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.026+0000 7fdd4940d740 1 -- 172.21.15.123:0/2451865739 wait complete. 2024-03-24T00:48:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 -- start start 2024-03-24T00:48:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 Processor -- start 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d6b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 0x558277d6b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d72880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277c82a20 con 0x558277d61bd0 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277cc28d0 con 0x558277d66d40 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x558277d69c90 con 0x558277d611e0 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd4940d740 10 client.0 _ll_register_callbacks cb 0x558277cbf1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.027+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d6b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d6b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:54900/0 (socket says 172.21.15.123:54900) 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3fe00640 1 -- 172.21.15.123:0/921790788 learned_addr learned my addr 172.21.15.123:0/921790788 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 0x558277d6b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d72880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 -- 172.21.15.123:0/921790788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 msgr2=0x558277d6b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d6b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 -- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 msgr2=0x558277d72880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d72880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 -- 172.21.15.123:0/921790788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582779db210 con 0x558277d61bd0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 0x558277d6b840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd4019b020 tx=0x7fdd2c012b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16082c728d1bc8d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.028+0000 7fdd29600640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c004020 con 0x558277d61bd0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdd2c0041c0 con 0x558277d61bd0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd38032c90 con 0x558277d61bd0 2024-03-24T00:48:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd2c0043c0 con 0x558277d61bd0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd4940d740 10 client.16425 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558277aa7620 con 0x558277d61bd0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd38032c90 con 0x558277d61bd0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.029+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd2c003ca0 con 0x558277d61bd0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 10 client.16425 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 20 client.16425 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277d772d0 0x558277d796f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 4 client.16425 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558277cd3d70 con 0x558277d772d0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277d772d0 0x558277d796f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277d772d0 0x558277d796f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feef275c12498de0 server_cookie=1198b38eba06680a in_seq=0 out_seq=0 2024-03-24T00:48:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.030+0000 7fdd29600640 10 client.16425 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd29600640 1 -- 172.21.15.123:0/921790788 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558277cd3d70 con 0x558277d772d0 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd29600640 10 client.16425 handle_command_reply: tid=0 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 client.16425 shutdown 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 msgr2=0x558277d6b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 0x558277d6b840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd4019b020 tx=0x7fdd2c012b40 comp rx=0 tx=0).stop 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 shutdown_connections 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d72880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d61bd0 0x558277d6b840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277d772d0 0x558277d796f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 --2- 172.21.15.123:0/921790788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d611e0 0x558277d6b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 >> 172.21.15.123:0/921790788 conn(0x558277d5c750 msgr2=0x558277d5b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.031+0000 7fdd29600640 0 client.16425 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.032+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 shutdown_connections 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.032+0000 7fdd4940d740 1 -- 172.21.15.123:0/921790788 wait complete. 2024-03-24T00:48:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.032+0000 7fdd4940d740 20 client.16425 trim_cache size 0 max 16384 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 Processor -- start 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 -- start start 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 0x558277d72840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401a8c40 con 0x558277d611e0 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401ab200 con 0x558277d66d40 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd4940d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401a9240 con 0x558277d61bd0 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57016/0 (socket says 172.21.15.123:57016) 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd3fe00640 1 -- 172.21.15.123:0/3583661583 learned_addr learned my addr 172.21.15.123:0/3583661583 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 0x558277d72840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.047+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 -- 172.21.15.123:0/3583661583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 msgr2=0x558277d732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 -- 172.21.15.123:0/3583661583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 msgr2=0x558277d575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 -- 172.21.15.123:0/3583661583 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558277d79d70 con 0x558277d61bd0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 0x558277d72840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009de0 tx=0x7fdd2c01c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fc19f9722a4c5a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd2be00640 1 -- 172.21.15.123:0/3583661583 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c01cb10 con 0x558277d61bd0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd2be00640 1 -- 172.21.15.123:0/3583661583 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c01ccb0 con 0x558277d61bd0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.048+0000 7fdd2be00640 1 -- 172.21.15.123:0/3583661583 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c00fd70 con 0x558277d61bd0 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- 172.21.15.123:0/3583661583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 msgr2=0x558277d72840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 0x558277d72840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009de0 tx=0x7fdd2c01c910 comp rx=0 tx=0).stop 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- 172.21.15.123:0/3583661583 shutdown_connections 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d66d40 0x558277d732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d61bd0 0x558277d72840 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 --2- 172.21.15.123:0/3583661583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- 172.21.15.123:0/3583661583 >> 172.21.15.123:0/3583661583 conn(0x558277d5c750 msgr2=0x558277d5f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- 172.21.15.123:0/3583661583 shutdown_connections 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- 172.21.15.123:0/3583661583 wait complete. 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.049+0000 7fdd4940d740 1 -- start start 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 Processor -- start 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d72180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d66d40 0x558277d70800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401a9880 con 0x558277d611e0 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401aaba0 con 0x558277d61bd0 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd401a9ee0 con 0x558277d66d40 2024-03-24T00:48:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd4940d740 10 client.0 _ll_register_callbacks cb 0x7fdd2c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd3ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd3fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d72180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd3ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:34008/0 (socket says 172.21.15.123:34008) 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd3ea00640 1 -- 172.21.15.123:0/4158150798 learned_addr learned my addr 172.21.15.123:0/4158150798 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.050+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d66d40 0x558277d70800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 -- 172.21.15.123:0/4158150798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d66d40 msgr2=0x558277d70800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d66d40 0x558277d70800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 -- 172.21.15.123:0/4158150798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 msgr2=0x558277d72180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d72180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 -- 172.21.15.123:0/4158150798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558277d70ef0 con 0x558277d61bd0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd3ea00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c01c4f0 tx=0x7fdd2c009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34474f669a3f200d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd29600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c01c6e0 con 0x558277d61bd0 2024-03-24T00:48:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdd2c00fc80 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd2c01ccb0 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd2c00fe80 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd4940d740 10 client.26606 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.051+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558277dd35a0 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c01ccb0 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd2c038030 con 0x558277d61bd0 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 10 client.26606 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 10 client.26606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 20 client.26606 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277dd3e80 0x558277dd62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 4 client.26606 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:48:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558277cce0c0 con 0x558277dd3e80 2024-03-24T00:48:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.052+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277dd3e80 0x558277dd62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.053+0000 7fdd3fe00640 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277dd3e80 0x558277dd62a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e85cbaca9e88319 server_cookie=1dbe38716fdec578 in_seq=0 out_seq=0 2024-03-24T00:48:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:13.053+0000 7fdd29600640 10 client.26606 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:14.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:14 smithi177 ceph-mon[36034]: pgmap v695: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-03-24T00:48:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:14 smithi204 ceph-mon[36345]: pgmap v695: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-03-24T00:48:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:14 smithi123 ceph-mon[31487]: pgmap v695: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-03-24T00:48:16.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:16 smithi177 ceph-mon[36034]: pgmap v696: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 34 KiB/s wr, 30 op/s 2024-03-24T00:48:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:16 smithi204 ceph-mon[36345]: pgmap v696: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 34 KiB/s wr, 30 op/s 2024-03-24T00:48:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:16 smithi123 ceph-mon[31487]: pgmap v696: 97 pgs: 97 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 34 KiB/s wr, 30 op/s 2024-03-24T00:48:17.773 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:17 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:17.798 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:17 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:17.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:17 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:18 smithi204 ceph-mon[36345]: pgmap v697: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 KiB/s wr, 19 op/s 2024-03-24T00:48:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:18 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:18 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:18 smithi123 ceph-mon[31487]: pgmap v697: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 KiB/s wr, 19 op/s 2024-03-24T00:48:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:18 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:18 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:18 smithi177 ceph-mon[36034]: pgmap v697: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 KiB/s wr, 19 op/s 2024-03-24T00:48:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:18 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:18.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:18 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:48:19.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.274+0000 7f891be00640 1 -- 172.21.15.123:0/3477329785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 msgr2=0x7f891c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.274+0000 7f891be00640 1 --2- 172.21.15.123:0/3477329785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c102c10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8908009900 tx=0x7f890802bc30 comp rx=0 tx=0).stop 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.274+0000 7f891be00640 1 -- 172.21.15.123:0/3477329785 shutdown_connections 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.274+0000 7f891be00640 1 --2- 172.21.15.123:0/3477329785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.275+0000 7f891be00640 1 --2- 172.21.15.123:0/3477329785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c102c10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.275+0000 7f891be00640 1 --2- 172.21.15.123:0/3477329785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.275+0000 7f891be00640 1 -- 172.21.15.123:0/3477329785 >> 172.21.15.123:0/3477329785 conn(0x7f891c0fcb10 msgr2=0x7f891c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.275+0000 7f891be00640 1 -- 172.21.15.123:0/3477329785 shutdown_connections 2024-03-24T00:48:19.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.275+0000 7f891be00640 1 -- 172.21.15.123:0/3477329785 wait complete. 2024-03-24T00:48:19.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 Processor -- start 2024-03-24T00:48:19.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 -- start start 2024-03-24T00:48:19.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c10d270 con 0x7f891c103150 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c10d0f0 con 0x7f891c102790 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c10d3f0 con 0x7f891c1015a0 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33780/0 (socket says 172.21.15.123:33780) 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.276+0000 7f891a000640 1 -- 172.21.15.123:0/569217904 learned_addr learned my addr 172.21.15.123:0/569217904 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891aa00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891b400640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 -- 172.21.15.123:0/569217904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 msgr2=0x7f891c197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 -- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 msgr2=0x7f891c19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 -- 172.21.15.123:0/569217904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c19f4b0 con 0x7f891c102790 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891aa00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891b400640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891a000640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8908000c00 tx=0x7f890802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f8913400640 1 -- 172.21.15.123:0/569217904 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908043020 con 0x7f891c102790 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f8913400640 1 -- 172.21.15.123:0/569217904 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89080054e0 con 0x7f891c102790 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f8913400640 1 -- 172.21.15.123:0/569217904 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908003c40 con 0x7f891c102790 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.277+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c19f740 con 0x7f891c102790 2024-03-24T00:48:19.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.278+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891c19fce0 con 0x7f891c102790 2024-03-24T00:48:19.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 Processor -- start 2024-03-24T00:48:19.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 -- start start 2024-03-24T00:48:19.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 0x5636bcc7f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc82040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcbdf250 con 0x5636bcc7a050 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcb87a30 con 0x5636bcc7aa40 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f89247e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcbdb2f0 con 0x5636bcc7fbb0 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f891b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f891b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48126/0 (socket says 172.21.15.123:48126) 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 0x5636bcc7f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f891b400640 1 -- 172.21.15.123:0/204441776 learned_addr learned my addr 172.21.15.123:0/204441776 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.282+0000 7f8913400640 1 -- 172.21.15.123:0/569217904 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8908046020 con 0x7f891c102790 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891a000640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc82040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 -- 172.21.15.123:0/204441776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 msgr2=0x5636bcc82040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc82040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 -- 172.21.15.123:0/204441776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 msgr2=0x5636bcc7a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 -- 172.21.15.123:0/204441776 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bcb72690 con 0x5636bcc7aa40 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891a000640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc82040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891b400640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.283+0000 7f891aa00640 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 0x5636bcc7f540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f890c009c00 tx=0x7f890c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37d6743d6179ceda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f8907e00640 1 -- 172.21.15.123:0/204441776 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c012450 con 0x5636bcc7aa40 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f8907e00640 1 -- 172.21.15.123:0/204441776 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f890c012ad0 con 0x5636bcc7aa40 2024-03-24T00:48:19.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f8907e00640 1 -- 172.21.15.123:0/204441776 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c00fb30 con 0x5636bcc7aa40 2024-03-24T00:48:19.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f8913400640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 0x7f89000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f8913400640 1 -- 172.21.15.123:0/569217904 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89080f0bb0 con 0x7f891c102790 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f891b400640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 0x7f89000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f89247e8740 1 -- 172.21.15.123:0/204441776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 msgr2=0x5636bcc7f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.284+0000 7f89247e8740 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 0x5636bcc7f540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f890c009c00 tx=0x7f890c0117b0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 -- 172.21.15.123:0/204441776 shutdown_connections 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc82040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7aa40 0x5636bcc7f540 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 --2- 172.21.15.123:0/204441776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7a050 0x5636bcc7a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 -- 172.21.15.123:0/204441776 >> 172.21.15.123:0/204441776 conn(0x5636bcc42200 msgr2=0x5636bcc759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 -- 172.21.15.123:0/204441776 shutdown_connections 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 -- 172.21.15.123:0/204441776 wait complete. 2024-03-24T00:48:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f891b400640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 0x7f89000a4d40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8914009e10 tx=0x7f8914009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 -- start start 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.285+0000 7f89247e8740 1 Processor -- start 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc845f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc8b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcc82b60 con 0x5636bcc7fbb0 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcb9ae80 con 0x5636bcc7a050 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcc82ce0 con 0x5636bcc7aa40 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f89247e8740 10 client.0 _ll_register_callbacks cb 0x5636bcbad130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc845f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33802/0 (socket says 172.21.15.123:33802) 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 -- 172.21.15.123:0/3153690535 learned_addr learned my addr 172.21.15.123:0/3153690535 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891b400640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc8b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 -- 172.21.15.123:0/3153690535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 msgr2=0x5636bcc845f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc845f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 -- 172.21.15.123:0/3153690535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 msgr2=0x5636bcc8b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc8b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.286+0000 7f891aa00640 1 -- 172.21.15.123:0/3153690535 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bc9bd7d0 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f891aa00640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f890c011cb0 tx=0x7f890c009f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a38f0bab7a7b0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f8904c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c029020 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f890c012760 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f890c012ad0 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f890c005040 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f89247e8740 10 client.26618 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.287+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5636bc8f33f0 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f890c012ad0 con 0x5636bcc7a050 2024-03-24T00:48:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f890c0046c0 con 0x5636bcc7a050 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 10 client.26618 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 10 client.26618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 20 client.26618 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcc8faf0 0x5636bcc91ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 4 client.26618 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636bcbec790 con 0x5636bcc8faf0 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.288+0000 7f891a000640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcc8faf0 0x5636bcc91ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.289+0000 7f8904c00640 10 client.26618 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:19.289+0000 7f891a000640 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcc8faf0 0x5636bcc91ee0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500d6625ff2651b2 server_cookie=df7f789c5594687b in_seq=0 out_seq=0 2024-03-24T00:48:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:19 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:19 smithi123 ceph-mon[31487]: pgmap v698: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-03-24T00:48:19.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:19 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:19.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:19 smithi177 ceph-mon[36034]: pgmap v698: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-03-24T00:48:20.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:20.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:20.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:20.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:19 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:20.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:19 smithi204 ceph-mon[36345]: pgmap v698: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-03-24T00:48:21.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:21 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:21 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.680+0000 7fdd29600640 1 -- 172.21.15.123:0/4158150798 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558277cce0c0 con 0x558277dd3e80 2024-03-24T00:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.680+0000 7fdd29600640 10 client.26606 handle_command_reply: tid=0 2024-03-24T00:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.680+0000 7fdd4940d740 1 client.26606 shutdown 2024-03-24T00:48:21.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.680+0000 7f8904c00640 1 -- 172.21.15.123:0/3153690535 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636bcbec790 con 0x5636bcc8faf0 2024-03-24T00:48:21.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.680+0000 7f8904c00640 10 client.26618 handle_command_reply: tid=0 2024-03-24T00:48:21.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 client.26618 shutdown 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 msgr2=0x558277d726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c01c4f0 tx=0x7fdd2c009280 comp rx=0 tx=0).stop 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 shutdown_connections 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x558277dd3e80 0x558277dd62a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558277d66d40 0x558277d70800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558277d61bd0 0x558277d726c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 --2- 172.21.15.123:0/4158150798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558277d611e0 0x558277d72180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 >> 172.21.15.123:0/4158150798 conn(0x558277d5c750 msgr2=0x558277d5f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd29600640 0 client.26606 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:21.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 msgr2=0x5636bcc840b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f890c011cb0 tx=0x7f890c009f20 comp rx=0 tx=0).stop 2024-03-24T00:48:21.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 shutdown_connections 2024-03-24T00:48:21.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc8b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcc8faf0 0x5636bcc91ee0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 shutdown_connections 2024-03-24T00:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.681+0000 7fdd4940d740 1 -- 172.21.15.123:0/4158150798 wait complete. 2024-03-24T00:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.682+0000 7fdd4940d740 20 client.26606 trim_cache size 0 max 16384 2024-03-24T00:48:21.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc845f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 --2- 172.21.15.123:0/3153690535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc840b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f8904c00640 0 client.26618 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 >> 172.21.15.123:0/3153690535 conn(0x5636bcc42200 msgr2=0x5636bcc74bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.681+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 shutdown_connections 2024-03-24T00:48:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.682+0000 7f89247e8740 1 -- 172.21.15.123:0/3153690535 wait complete. 2024-03-24T00:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:48:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.682+0000 7f89247e8740 20 client.26618 trim_cache size 0 max 16384 2024-03-24T00:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.682+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 msgr2=0x7fdd240a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.682+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 0x7fdd240a4d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdd300cf100 tx=0x7fdd300d28a0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.682+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 msgr2=0x7fdd40199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.682+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 shutdown_connections 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd401054b0 0x7fdd401a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdd240a28a0 0x7fdd240a4d60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd40104af0 0x7fdd4019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 --2- 172.21.15.123:0/833365964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd40103900 0x7fdd40199b10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 >> 172.21.15.123:0/833365964 conn(0x7fdd400fee90 msgr2=0x7fdd40107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 shutdown_connections 2024-03-24T00:48:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:21.683+0000 7fdd46600640 1 -- 172.21.15.123:0/833365964 wait complete. 2024-03-24T00:48:21.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:21 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:21.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.714+0000 7f89247e8740 1 Processor -- start 2024-03-24T00:48:21.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.714+0000 7f89247e8740 1 -- start start 2024-03-24T00:48:21.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.714+0000 7f89247e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f89247e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc8b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f89247e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc96750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f89247e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a67f0 con 0x5636bcc7fbb0 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f89247e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a8db0 con 0x5636bcc7a050 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f89247e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a6df0 con 0x5636bcc7aa40 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33806/0 (socket says 172.21.15.123:33806) 2024-03-24T00:48:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 -- 172.21.15.123:0/3725535909 learned_addr learned my addr 172.21.15.123:0/3725535909 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891a000640 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc8b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891b400640 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc96750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 -- 172.21.15.123:0/3725535909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 msgr2=0x5636bcc8b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc8b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 -- 172.21.15.123:0/3725535909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 msgr2=0x5636bcc96750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc96750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.715+0000 7f891aa00640 1 -- 172.21.15.123:0/3725535909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bc8b30f0 con 0x5636bcc7a050 2024-03-24T00:48:21.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.716+0000 7f891aa00640 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f890c009b00 tx=0x7f890c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ba04c278353fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.716+0000 7f8907400640 1 -- 172.21.15.123:0/3725535909 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c03e070 con 0x5636bcc7a050 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.716+0000 7f8907400640 1 -- 172.21.15.123:0/3725535909 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f890c009720 con 0x5636bcc7a050 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.716+0000 7f8907400640 1 -- 172.21.15.123:0/3725535909 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c029040 con 0x5636bcc7a050 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 -- 172.21.15.123:0/3725535909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 msgr2=0x5636bcc7bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f890c009b00 tx=0x7f890c0117b0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 -- 172.21.15.123:0/3725535909 shutdown_connections 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7fbb0 0x5636bcc96750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7aa40 0x5636bcc8b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 --2- 172.21.15.123:0/3725535909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc7bf80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 -- 172.21.15.123:0/3725535909 >> 172.21.15.123:0/3725535909 conn(0x5636bcc42200 msgr2=0x5636bcc74bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 -- 172.21.15.123:0/3725535909 shutdown_connections 2024-03-24T00:48:21.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.717+0000 7f89247e8740 1 -- 172.21.15.123:0/3725535909 wait complete. 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 -- start start 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 Processor -- start 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 0x5636bcc89010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc89fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a7430 con 0x5636bcc7aa40 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a8750 con 0x5636bcc7a050 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1a7a90 con 0x5636bcc7fbb0 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f89247e8740 10 client.0 _ll_register_callbacks cb 0x7f8914009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.718+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33812/0 (socket says 172.21.15.123:33812) 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 -- 172.21.15.123:0/4266396568 learned_addr learned my addr 172.21.15.123:0/4266396568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891b400640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc89fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891a000640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 0x5636bcc89010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 -- 172.21.15.123:0/4266396568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 msgr2=0x5636bcc89fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc89fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 -- 172.21.15.123:0/4266396568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 msgr2=0x5636bcc89010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 0x5636bcc89010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 -- 172.21.15.123:0/4266396568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bcc8a5c0 con 0x5636bcc7a050 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891a000640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 0x5636bcc89010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f891aa00640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f890c03c040 tx=0x7f890c012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b5a13d02032b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.719+0000 7f8904c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c03e070 con 0x5636bcc7a050 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f890c004540 con 0x5636bcc7a050 2024-03-24T00:48:21.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f890c009720 con 0x5636bcc7a050 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f890c029040 con 0x5636bcc7a050 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 10 client.26624 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5636bcc8b8d0 con 0x5636bcc7a050 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f890c009720 con 0x5636bcc7a050 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f890c037070 con 0x5636bcc7a050 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 10 client.26624 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 10 client.26624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 20 client.26624 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcceb800 0x5636bccedc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 4 client.26624 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.720+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5636bcbe6ae0 con 0x5636bcceb800 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f891a000640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcceb800 0x5636bccedc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f891a000640 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcceb800 0x5636bccedc20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1363c567aa4fb6 server_cookie=f421c90edbaa97e4 in_seq=0 out_seq=0 2024-03-24T00:48:21.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f8904c00640 10 client.26624 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:21.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f8904c00640 1 -- 172.21.15.123:0/4266396568 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5636bcbe6ae0 con 0x5636bcceb800 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f8904c00640 10 client.26624 handle_command_reply: tid=0 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.721+0000 7f89247e8740 1 client.26624 shutdown 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 msgr2=0x5636bcc88ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f890c03c040 tx=0x7f890c012990 comp rx=0 tx=0).stop 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 shutdown_connections 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5636bcceb800 0x5636bccedc20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636bcc7fbb0 0x5636bcc89fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5636bcc7aa40 0x5636bcc89010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 --2- 172.21.15.123:0/4266396568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636bcc7a050 0x5636bcc88ad0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 >> 172.21.15.123:0/4266396568 conn(0x5636bcc42200 msgr2=0x5636bcc78c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f8904c00640 0 client.26624 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 shutdown_connections 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 1 -- 172.21.15.123:0/4266396568 wait complete. 2024-03-24T00:48:21.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.722+0000 7f89247e8740 20 client.26624 trim_cache size 0 max 16384 2024-03-24T00:48:21.727 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:21.727 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:48:21.727 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:48:21.727 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 msgr2=0x7f89000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 0x7f89000a4d40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8914009e10 tx=0x7f8914009500 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 msgr2=0x7f891c197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8908000c00 tx=0x7f890802c650 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 shutdown_connections 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f89000a2880 0x7f89000a4d40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f891c103150 0x7f891c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f891c102790 0x7f891c197dd0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 --2- 172.21.15.123:0/569217904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f891c1015a0 0x7f891c197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 >> 172.21.15.123:0/569217904 conn(0x7f891c0fcb10 msgr2=0x7f891c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 shutdown_connections 2024-03-24T00:48:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.723+0000 7f891be00640 1 -- 172.21.15.123:0/569217904 wait complete. 2024-03-24T00:48:21.742 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:48:21.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 -- 172.21.15.123:0/3168532688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0102130 msgr2=0x7fb4b0104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 --2- 172.21.15.123:0/3168532688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0102130 0x7fb4b0104520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009900 tx=0x7fb4a002bc30 comp rx=0 tx=0).stop 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 -- 172.21.15.123:0/3168532688 shutdown_connections 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 --2- 172.21.15.123:0/3168532688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 0x7fb4b01097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 --2- 172.21.15.123:0/3168532688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0104a60 0x7fb4b0106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 --2- 172.21.15.123:0/3168532688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0102130 0x7fb4b0104520 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.933+0000 7fb4b6400640 1 -- 172.21.15.123:0/3168532688 >> 172.21.15.123:0/3168532688 conn(0x7fb4b00fbb20 msgr2=0x7fb4b00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.934+0000 7fb4b6400640 1 -- 172.21.15.123:0/3168532688 shutdown_connections 2024-03-24T00:48:21.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.934+0000 7fb4b6400640 1 -- 172.21.15.123:0/3168532688 wait complete. 2024-03-24T00:48:21.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.934+0000 7fb4b6400640 1 Processor -- start 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 -- start start 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 0x7fb4b0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 0x7fb4b019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b010d3a0 con 0x7fb4b0102130 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b010d220 con 0x7fb4b0107390 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4b6400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b010d520 con 0x7fb4b0104a60 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4af400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4af400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48178/0 (socket says 172.21.15.123:48178) 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4af400640 1 -- 172.21.15.123:0/2866716144 learned_addr learned my addr 172.21.15.123:0/2866716144 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4aea00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 0x7fb4b0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.935+0000 7fb4afe00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 0x7fb4b019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 -- 172.21.15.123:0/2866716144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 msgr2=0x7fb4b0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 0x7fb4b0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 -- 172.21.15.123:0/2866716144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 msgr2=0x7fb4b019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 0x7fb4b019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 -- 172.21.15.123:0/2866716144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b019f5f0 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4aea00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 0x7fb4b0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4af400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0000c00 tx=0x7fb4a002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0043020 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4a00054e0 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b00fff20 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0003c40 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.936+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b01004c0 con 0x7fb4b0102130 2024-03-24T00:48:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.938+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b00fd610 con 0x7fb4b0102130 2024-03-24T00:48:21.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.940+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb4a0046020 con 0x7fb4b0102130 2024-03-24T00:48:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.941+0000 7fb4a7e00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 0x7fb4940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.941+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb4a00f0aa0 con 0x7fb4b0102130 2024-03-24T00:48:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.941+0000 7fb4aea00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 0x7fb4940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.942+0000 7fb4aea00640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 0x7fb4940a4d40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0095f0 tx=0x7fb49c012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:21.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:21.946+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb4a004d020 con 0x7fb4b0102130 2024-03-24T00:48:22.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.202+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4b002d070 con 0x7fb4b0102130 2024-03-24T00:48:22.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.203+0000 7fb4a7e00640 1 -- 172.21.15.123:0/2866716144 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fb4a00bdaf0 con 0x7fb4b0102130 2024-03-24T00:48:22.204 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:48:22.204 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:48:22.204 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:48:22.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 msgr2=0x7fb4940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 0x7fb4940a4d40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0095f0 tx=0x7fb49c012040 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 msgr2=0x7fb4b01979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0000c00 tx=0x7fb4a002c650 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 shutdown_connections 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4b0107390 0x7fb4b019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4b0104a60 0x7fb4b0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fb4940a2880 0x7fb4940a4d40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 --2- 172.21.15.123:0/2866716144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4b0102130 0x7fb4b01979d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 >> 172.21.15.123:0/2866716144 conn(0x7fb4b00fbb20 msgr2=0x7fb4b0102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 shutdown_connections 2024-03-24T00:48:22.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.206+0000 7fb4b6400640 1 -- 172.21.15.123:0/2866716144 wait complete. 2024-03-24T00:48:22.227 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:48:22.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:48:22.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 -- 172.21.15.123:0/1923917846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa200105490 msgr2=0x7fa20010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 --2- 172.21.15.123:0/1923917846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa200105490 0x7fa20010bd20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009ca0 tx=0x7fa1f802c100 comp rx=0 tx=0).stop 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 -- 172.21.15.123:0/1923917846 shutdown_connections 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 --2- 172.21.15.123:0/1923917846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa200105490 0x7fa20010bd20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 --2- 172.21.15.123:0/1923917846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200104ad0 0x7fa200104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 --2- 172.21.15.123:0/1923917846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2001038e0 0x7fa200103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 -- 172.21.15.123:0/1923917846 >> 172.21.15.123:0/1923917846 conn(0x7fa2000fee50 msgr2=0x7fa200101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 -- 172.21.15.123:0/1923917846 shutdown_connections 2024-03-24T00:48:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.423+0000 7fa208400640 1 -- 172.21.15.123:0/1923917846 wait complete. 2024-03-24T00:48:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.424+0000 7fa208400640 1 Processor -- start 2024-03-24T00:48:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 -- start start 2024-03-24T00:48:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 0x7fa200199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20010f540 con 0x7fa2001038e0 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20010f3c0 con 0x7fa200105490 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa208400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20010f6c0 con 0x7fa200104ad0 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa205000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa206400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa205000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46804/0 (socket says 172.21.15.123:46804) 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa206400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33836/0 (socket says 172.21.15.123:33836) 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa205000640 1 -- 172.21.15.123:0/3370528176 learned_addr learned my addr 172.21.15.123:0/3370528176 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.425+0000 7fa205a00640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 0x7fa200199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 -- 172.21.15.123:0/3370528176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 msgr2=0x7fa20019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 -- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 msgr2=0x7fa200199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 0x7fa200199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 -- 172.21.15.123:0/3370528176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2001a1780 con 0x7fa200105490 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa205a00640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 0x7fa200199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa205000640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa206400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8037040 tx=0x7fa1f8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa1fe000640 1 -- 172.21.15.123:0/3370528176 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8039040 con 0x7fa200105490 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa1fe000640 1 -- 172.21.15.123:0/3370528176 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f8044070 con 0x7fa200105490 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2001a1a10 con 0x7fa200105490 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.426+0000 7fa1fe000640 1 -- 172.21.15.123:0/3370528176 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8004350 con 0x7fa200105490 2024-03-24T00:48:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.427+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2001a1f20 con 0x7fa200105490 2024-03-24T00:48:22.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa1fe000640 1 -- 172.21.15.123:0/3370528176 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa1f80044f0 con 0x7fa200105490 2024-03-24T00:48:22.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 Processor -- start 2024-03-24T00:48:22.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 -- start start 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dc0562c0 con 0x55c8dc0f29e0 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dbffec90 con 0x55c8dc0f33a0 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa20b1da740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dc052360 con 0x55c8dc0f1810 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa205a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.431+0000 7fa205000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48204/0 (socket says 172.21.15.123:48204) 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 -- 172.21.15.123:0/2342385256 learned_addr learned my addr 172.21.15.123:0/2342385256 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205000640 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 -- 172.21.15.123:0/2342385256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 msgr2=0x55c8dc0f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 -- 172.21.15.123:0/2342385256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 msgr2=0x55c8dc0f9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 -- 172.21.15.123:0/2342385256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8dc023080 con 0x55c8dc0f29e0 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa1fe000640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa205a00640 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009cf0 tx=0x7fa1f40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db6ae85b21e7bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.432+0000 7fa1fe000640 1 -- 172.21.15.123:0/3370528176 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1f80f0ff0 con 0x7fa200105490 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa205000640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa1f2a00640 1 -- 172.21.15.123:0/2342385256 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012450 con 0x55c8dc0f29e0 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa1f2a00640 1 -- 172.21.15.123:0/2342385256 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f4012ad0 con 0x55c8dc0f29e0 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa1f2a00640 1 -- 172.21.15.123:0/2342385256 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f400fbf0 con 0x55c8dc0f29e0 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 -- 172.21.15.123:0/2342385256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 msgr2=0x55c8dc0f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009cf0 tx=0x7fa1f40117b0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa205000640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 0x7fa1e80a4d40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55c8dc0f28e0 tx=0x7fa1ec0046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 -- 172.21.15.123:0/2342385256 shutdown_connections 2024-03-24T00:48:22.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f2e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 --2- 172.21.15.123:0/2342385256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.433+0000 7fa20b1da740 1 -- 172.21.15.123:0/2342385256 >> 172.21.15.123:0/2342385256 conn(0x55c8dc0ecc00 msgr2=0x55c8dc0ed000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.434+0000 7fa20b1da740 1 -- 172.21.15.123:0/2342385256 shutdown_connections 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.434+0000 7fa20b1da740 1 -- 172.21.15.123:0/2342385256 wait complete. 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.434+0000 7fa20b1da740 1 -- start start 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.434+0000 7fa20b1da740 1 Processor -- start 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc103030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dc0fa3b0 con 0x55c8dc0f33a0 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dc0120d0 con 0x55c8dc0f29e0 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8dc0fa530 con 0x55c8dc0f1810 2024-03-24T00:48:22.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa20b1da740 10 client.0 _ll_register_callbacks cb 0x55c8dc016e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46820/0 (socket says 172.21.15.123:46820) 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 -- 172.21.15.123:0/3716640416 learned_addr learned my addr 172.21.15.123:0/3716640416 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa206400640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 -- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f29e0 msgr2=0x55c8dc0fbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 -- 172.21.15.123:0/3716640416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 msgr2=0x55c8dc103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 -- 172.21.15.123:0/3716640416 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8dbd6a3f0 con 0x55c8dc0f1810 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa206400640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.435+0000 7fa205a00640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa1f401c040 tx=0x7fa1f4009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=568e0081af500af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa1e7400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4029030 con 0x55c8dc0f1810 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1f4020040 con 0x55c8dc0f1810 2024-03-24T00:48:22.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1f4012ad0 con 0x55c8dc0f1810 2024-03-24T00:48:22.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1f40126e0 con 0x55c8dc0f1810 2024-03-24T00:48:22.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa20b1da740 10 client.26647 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:22.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c8dbe347d0 con 0x55c8dc0f1810 2024-03-24T00:48:22.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.436+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f4012ad0 con 0x55c8dc0f1810 2024-03-24T00:48:22.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa1f40054d0 con 0x55c8dc0f1810 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 10 client.26647 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 10 client.26647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 20 client.26647 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc1072a0 0x55c8dc109690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 4 client.26647 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8dc063800 con 0x55c8dc1072a0 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.437+0000 7fa205000640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc1072a0 0x55c8dc109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.438+0000 7fa205000640 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc1072a0 0x55c8dc109690 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7841c9ac6d62e1 server_cookie=90e309a78f93e114 in_seq=0 out_seq=0 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.438+0000 7fa1e7400640 10 client.26647 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.438+0000 7fa1e7400640 1 -- 172.21.15.123:0/3716640416 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c8dc063800 con 0x55c8dc1072a0 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.438+0000 7fa1e7400640 10 client.26647 handle_command_reply: tid=0 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 client.26647 shutdown 2024-03-24T00:48:22.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 msgr2=0x55c8dc0fbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa1f401c040 tx=0x7fa1f4009a20 comp rx=0 tx=0).stop 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 shutdown_connections 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc1072a0 0x55c8dc109690 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 --2- 172.21.15.123:0/3716640416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fbab0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa1e7400640 0 client.26647 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 >> 172.21.15.123:0/3716640416 conn(0x55c8dc0ecc00 msgr2=0x55c8dc0ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 shutdown_connections 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 1 -- 172.21.15.123:0/3716640416 wait complete. 2024-03-24T00:48:22.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.439+0000 7fa20b1da740 20 client.26647 trim_cache size 0 max 16384 2024-03-24T00:48:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 Processor -- start 2024-03-24T00:48:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 -- start start 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001a8c70 con 0x55c8dc0f1810 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001ab230 con 0x55c8dc0f33a0 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa20b1da740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001a9270 con 0x55c8dc0f29e0 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa205a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa206400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa205a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48220/0 (socket says 172.21.15.123:48220) 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa205a00640 1 -- 172.21.15.123:0/936920960 learned_addr learned my addr 172.21.15.123:0/936920960 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.453+0000 7fa205000640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 -- 172.21.15.123:0/936920960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 msgr2=0x55c8dc0f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 -- 172.21.15.123:0/936920960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 msgr2=0x55c8dc0f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 -- 172.21.15.123:0/936920960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8dc0eda80 con 0x55c8dc0f1810 2024-03-24T00:48:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205000640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa206400640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa205a00640 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e60 tx=0x7fa1f40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516c81d38fce1bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa1f2000640 1 -- 172.21.15.123:0/936920960 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4029040 con 0x55c8dc0f1810 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa1f2000640 1 -- 172.21.15.123:0/936920960 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f400f440 con 0x55c8dc0f1810 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.454+0000 7fa1f2000640 1 -- 172.21.15.123:0/936920960 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f400fa30 con 0x55c8dc0f1810 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- 172.21.15.123:0/936920960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 msgr2=0x55c8dc0f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e60 tx=0x7fa1f40117b0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- 172.21.15.123:0/936920960 shutdown_connections 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f33a0 0x55c8dc0f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 --2- 172.21.15.123:0/936920960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f1810 0x55c8dc0f59e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- 172.21.15.123:0/936920960 >> 172.21.15.123:0/936920960 conn(0x55c8dc0ecc00 msgr2=0x55c8dc0f06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- 172.21.15.123:0/936920960 shutdown_connections 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- 172.21.15.123:0/936920960 wait complete. 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 -- start start 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.455+0000 7fa20b1da740 1 Processor -- start 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fe1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc1011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001a98b0 con 0x55c8dc0f33a0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001aabd0 con 0x55c8dc0f1810 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001a9f10 con 0x55c8dc0f29e0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa20b1da740 10 client.0 _ll_register_callbacks cb 0x7fa1f4009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fe1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33884/0 (socket says 172.21.15.123:33884) 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 -- 172.21.15.123:0/635712325 learned_addr learned my addr 172.21.15.123:0/635712325 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa206400640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc1011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 -- 172.21.15.123:0/635712325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 msgr2=0x55c8dc0fe1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fe1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 -- 172.21.15.123:0/635712325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 msgr2=0x55c8dc1011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc1011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205a00640 1 -- 172.21.15.123:0/635712325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8dc1017c0 con 0x55c8dc0f1810 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa206400640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.456+0000 7fa205000640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa205a00640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009ef0 tx=0x7fa1f40094d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2a566f7a7ad6e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa1e7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4045060 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1f400f750 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8dc05db50 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1f4029040 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa20b1da740 10 client.26648 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c8dc102ac0 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.457+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c8dc05db50 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa1f4005340 con 0x55c8dc0f1810 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 10 client.26648 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 10 client.26648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 20 client.26648 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc162ff0 0x55c8dc1653e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 4 client.26648 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:48:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c8dbfdb500 con 0x55c8dc162ff0 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa205000640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc162ff0 0x55c8dc1653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa205000640 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc162ff0 0x55c8dc1653e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40011e5792dad40b server_cookie=d1a9456bb977bcc5 in_seq=0 out_seq=0 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.458+0000 7fa1e7400640 10 client.26648 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa1e7400640 1 -- 172.21.15.123:0/635712325 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c8dbfdb500 con 0x55c8dc162ff0 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa1e7400640 10 client.26648 handle_command_reply: tid=0 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 client.26648 shutdown 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 msgr2=0x55c8dc0fdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009ef0 tx=0x7fa1f40094d0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 shutdown_connections 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8dc0f33a0 0x55c8dc1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55c8dc162ff0 0x55c8dc1653e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8dc0f29e0 0x55c8dc0fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 --2- 172.21.15.123:0/635712325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8dc0f1810 0x55c8dc0fdc70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa1e7400640 0 client.26648 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:22.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 >> 172.21.15.123:0/635712325 conn(0x55c8dc0ecc00 msgr2=0x55c8dc0eb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 shutdown_connections 2024-03-24T00:48:22.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 1 -- 172.21.15.123:0/635712325 wait complete. 2024-03-24T00:48:22.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.459+0000 7fa20b1da740 20 client.26648 trim_cache size 0 max 16384 2024-03-24T00:48:22.465 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 msgr2=0x7fa1e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 0x7fa1e80a4d40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55c8dc0f28e0 tx=0x7fa1ec0046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 msgr2=0x7fa2001a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8037040 tx=0x7fa1f8035d10 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 shutdown_connections 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa200105490 0x7fa2001a1110 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa200104ad0 0x7fa20019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 --2- 172.21.15.123:0/3370528176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2001038e0 0x7fa200199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 >> 172.21.15.123:0/3370528176 conn(0x7fa2000fee50 msgr2=0x7fa200107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 shutdown_connections 2024-03-24T00:48:22.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:22.460+0000 7fa208400640 1 -- 172.21.15.123:0/3370528176 wait complete. 2024-03-24T00:48:22.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:22 smithi123 ceph-mon[31487]: pgmap v699: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 20 op/s 2024-03-24T00:48:22.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:22.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:22.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:22.477 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:22 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2866716144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:22 smithi204 ceph-mon[36345]: pgmap v699: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 20 op/s 2024-03-24T00:48:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:22.571 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:22 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2866716144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:22.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:22 smithi177 ceph-mon[36034]: pgmap v699: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 KiB/s wr, 20 op/s 2024-03-24T00:48:22.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:22.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:22.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:48:22.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:22 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2866716144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:23.481 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:48:23.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:23.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 -- 172.21.15.123:0/3259784777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64102130 msgr2=0x7f2e64104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3259784777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64102130 0x7f2e64104520 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009900 tx=0x7f2e5802bc40 comp rx=0 tx=0).stop 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 -- 172.21.15.123:0/3259784777 shutdown_connections 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3259784777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64107390 0x7f2e641097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3259784777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64104a60 0x7f2e64106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3259784777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64102130 0x7f2e64104520 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 -- 172.21.15.123:0/3259784777 >> 172.21.15.123:0/3259784777 conn(0x7f2e640fbb40 msgr2=0x7f2e640fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 -- 172.21.15.123:0/3259784777 shutdown_connections 2024-03-24T00:48:23.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.704+0000 7f2e6c400640 1 -- 172.21.15.123:0/3259784777 wait complete. 2024-03-24T00:48:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.705+0000 7f2e6c400640 1 Processor -- start 2024-03-24T00:48:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.705+0000 7f2e6c400640 1 -- start start 2024-03-24T00:48:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 0x7f2e6419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410d380 con 0x7f2e64107390 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410d200 con 0x7f2e64104a60 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6c400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6410d500 con 0x7f2e64102130 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 0x7f2e6419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33904/0 (socket says 172.21.15.123:33904) 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46864/0 (socket says 172.21.15.123:46864) 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 0x7f2e6419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48252/0 (socket says 172.21.15.123:48252) 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.706+0000 7f2e69000640 1 -- 172.21.15.123:0/3006243034 learned_addr learned my addr 172.21.15.123:0/3006243034 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 -- 172.21.15.123:0/3006243034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 msgr2=0x7f2e64197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 -- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 msgr2=0x7f2e6419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 0x7f2e6419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 -- 172.21.15.123:0/3006243034 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6419f690 con 0x7f2e64104a60 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e69000640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000e9c0 tx=0x7f2e5000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e62000640 1 -- 172.21.15.123:0/3006243034 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000ccc0 con 0x7f2e64104a60 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e62000640 1 -- 172.21.15.123:0/3006243034 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e50005500 con 0x7f2e64104a60 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e62000640 1 -- 172.21.15.123:0/3006243034 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50010430 con 0x7f2e64104a60 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6419f980 con 0x7f2e64104a60 2024-03-24T00:48:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.707+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6419fe80 con 0x7f2e64104a60 2024-03-24T00:48:23.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.711+0000 7f2e6f177740 1 Processor -- start 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 -- start start 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e62000640 1 -- 172.21.15.123:0/3006243034 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2e50005000 con 0x7f2e64104a60 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b7010 0x5623871b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562387112eb0 con 0x5623871b14b0 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623870bba60 con 0x5623871b1ea0 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6f177740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56238710ef50 con 0x5623871b7010 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33906/0 (socket says 172.21.15.123:33906) 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e69a00640 1 -- 172.21.15.123:0/3022039103 learned_addr learned my addr 172.21.15.123:0/3022039103 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e6a400640 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.712+0000 7f2e69000640 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b7010 0x5623871b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 -- 172.21.15.123:0/3022039103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b7010 msgr2=0x5623871b94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b7010 0x5623871b94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 -- 172.21.15.123:0/3022039103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 msgr2=0x5623871b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 -- 172.21.15.123:0/3022039103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562386eef7e0 con 0x5623871b1ea0 2024-03-24T00:48:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005ab0 tx=0x7f2e58000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad96e9e86777fb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e56a00640 1 -- 172.21.15.123:0/3022039103 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58042070 con 0x5623871b1ea0 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e56a00640 1 -- 172.21.15.123:0/3022039103 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e5802cd40 con 0x5623871b1ea0 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e56a00640 1 -- 172.21.15.123:0/3022039103 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58005270 con 0x5623871b1ea0 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.713+0000 7f2e62000640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 0x7f2e4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e62000640 1 -- 172.21.15.123:0/3006243034 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2e500cee60 con 0x7f2e64104a60 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6a400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 0x7f2e4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 -- 172.21.15.123:0/3022039103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 msgr2=0x5623871b69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005ab0 tx=0x7f2e58000bc0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 -- 172.21.15.123:0/3022039103 shutdown_connections 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b7010 0x5623871b94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871b69a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3022039103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 -- 172.21.15.123:0/3022039103 >> 172.21.15.123:0/3022039103 conn(0x5623871a8610 msgr2=0x5623871a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 -- 172.21.15.123:0/3022039103 shutdown_connections 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.714+0000 7f2e6f177740 1 -- 172.21.15.123:0/3022039103 wait complete. 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.715+0000 7f2e6a400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 0x7f2e4c0a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x5623871b2700 tx=0x7f2e5c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.715+0000 7f2e6f177740 1 -- start start 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.715+0000 7f2e6f177740 1 Processor -- start 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b1ea0 0x5623871bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 0x5623871c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623871b9f60 con 0x5623871b14b0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623870ceeb0 con 0x5623871b7010 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623871ba0e0 con 0x5623871b1ea0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6f177740 10 client.0 _ll_register_callbacks cb 0x5623870c3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 0x5623871c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48272/0 (socket says 172.21.15.123:48272) 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 -- 172.21.15.123:0/172090039 learned_addr learned my addr 172.21.15.123:0/172090039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69000640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b1ea0 0x5623871bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 -- 172.21.15.123:0/172090039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b1ea0 msgr2=0x5623871bbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b1ea0 0x5623871bbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 -- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 msgr2=0x5623871c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 0x5623871c2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e69a00640 1 -- 172.21.15.123:0/172090039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562386f4ff70 con 0x5623871b14b0 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.716+0000 7f2e6a400640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 0x5623871c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e69a00640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2e5802c250 tx=0x7f2e580056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a55f5db59b678d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e4b400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58042040 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2e58004b90 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e5802cd40 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e58047460 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e6f177740 10 client.16482 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.717+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562386de7170 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e5802cd40 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2e580504e0 con 0x5623871b14b0 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 10 client.16482 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 20 client.16482 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623871c7080 0x5623871c9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 4 client.16482 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623871203f0 con 0x5623871c7080 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.718+0000 7f2e69000640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623871c7080 0x5623871c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.719+0000 7f2e69000640 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623871c7080 0x5623871c9470 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bcf394094353b28 server_cookie=d55697f87acf82a7 in_seq=0 out_seq=0 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.719+0000 7f2e4b400640 10 client.16482 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.719+0000 7f2e4b400640 1 -- 172.21.15.123:0/172090039 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5623871203f0 con 0x5623871c7080 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e4b400640 10 client.16482 handle_command_reply: tid=0 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 client.16482 shutdown 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 msgr2=0x5623871bb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2e5802c250 tx=0x7f2e580056b0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 shutdown_connections 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b7010 0x5623871c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623871c7080 0x5623871c9470 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b1ea0 0x5623871bbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 --2- 172.21.15.123:0/172090039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b14b0 0x5623871bb610 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e4b400640 0 client.16482 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 >> 172.21.15.123:0/172090039 conn(0x5623871a8610 msgr2=0x5623871a7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 shutdown_connections 2024-03-24T00:48:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 1 -- 172.21.15.123:0/172090039 wait complete. 2024-03-24T00:48:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.720+0000 7f2e6f177740 20 client.16482 trim_cache size 0 max 16384 2024-03-24T00:48:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 Processor -- start 2024-03-24T00:48:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 -- start start 2024-03-24T00:48:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a48a0 con 0x5623871b7010 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a6e60 con 0x5623871b1ea0 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.739+0000 7f2e6f177740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a4ea0 con 0x5623871b14b0 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46890/0 (socket says 172.21.15.123:46890) 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871a7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33926/0 (socket says 172.21.15.123:33926) 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 -- 172.21.15.123:0/3885007544 learned_addr learned my addr 172.21.15.123:0/3885007544 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 -- 172.21.15.123:0/3885007544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 msgr2=0x5623871a7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871a7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 -- 172.21.15.123:0/3885007544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 msgr2=0x5623871cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 -- 172.21.15.123:0/3885007544 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623871a9380 con 0x5623871b14b0 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e69a00640 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e5802c180 tx=0x7f2e580097c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d28fee2c866a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.740+0000 7f2e56000640 1 -- 172.21.15.123:0/3885007544 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58042070 con 0x5623871b14b0 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e56000640 1 -- 172.21.15.123:0/3885007544 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e5802cd40 con 0x5623871b14b0 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e56000640 1 -- 172.21.15.123:0/3885007544 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58047600 con 0x5623871b14b0 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 -- 172.21.15.123:0/3885007544 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 msgr2=0x5623871a76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e5802c180 tx=0x7f2e580097c0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 -- 172.21.15.123:0/3885007544 shutdown_connections 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 --2- 172.21.15.123:0/3885007544 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871a76b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 -- 172.21.15.123:0/3885007544 >> 172.21.15.123:0/3885007544 conn(0x5623871a8610 msgr2=0x5623871a6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 -- 172.21.15.123:0/3885007544 shutdown_connections 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.741+0000 7f2e6f177740 1 -- 172.21.15.123:0/3885007544 wait complete. 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 -- start start 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 Processor -- start 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a54e0 con 0x5623871b7010 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a6800 con 0x5623871b1ea0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641a5b40 con 0x5623871b14b0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e6f177740 10 client.0 _ll_register_callbacks cb 0x7f2e5c009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.742+0000 7f2e69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33938/0 (socket says 172.21.15.123:33938) 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 -- 172.21.15.123:0/1556963174 learned_addr learned my addr 172.21.15.123:0/1556963174 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69a00640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e6a400640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 -- 172.21.15.123:0/1556963174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 msgr2=0x5623871bf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871bf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 -- 172.21.15.123:0/1556963174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 msgr2=0x5623871c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 -- 172.21.15.123:0/1556963174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623871c1130 con 0x5623871b1ea0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e69000640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2e500cc970 tx=0x7f2e50086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b589055798246e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e4b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5009b170 con 0x5623871b1ea0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2e50099b10 con 0x5623871b1ea0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e5802cd40 con 0x5623871b1ea0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.743+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e5009bf10 con 0x5623871b1ea0 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 10 client.26666 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:23.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562387222d40 con 0x5623871b1ea0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e5802cd40 con 0x5623871b1ea0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2e500860b0 con 0x5623871b1ea0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 10 client.26666 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 10 client.26666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 20 client.26666 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623872236a0 0x562387225ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 4 client.26666 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56238711a740 con 0x5623872236a0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.744+0000 7f2e69a00640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623872236a0 0x562387225ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.745+0000 7f2e69a00640 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623872236a0 0x562387225ac0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c92e3620cd11a5be server_cookie=5bcdc2f85440ea19 in_seq=0 out_seq=0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.745+0000 7f2e4b400640 10 client.26666 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e4b400640 1 -- 172.21.15.123:0/1556963174 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56238711a740 con 0x5623872236a0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e4b400640 10 client.26666 handle_command_reply: tid=0 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 client.26666 shutdown 2024-03-24T00:48:23.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 msgr2=0x5623871bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2e500cc970 tx=0x7f2e50086040 comp rx=0 tx=0).stop 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 shutdown_connections 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623871b7010 0x5623871c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623871b1ea0 0x5623871bfb30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x5623872236a0 0x562387225ac0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 --2- 172.21.15.123:0/1556963174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5623871b14b0 0x5623871bf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e4b400640 0 client.26666 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 >> 172.21.15.123:0/1556963174 conn(0x5623871a8610 msgr2=0x5623871a7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 shutdown_connections 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.746+0000 7f2e6f177740 1 -- 172.21.15.123:0/1556963174 wait complete. 2024-03-24T00:48:23.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6f177740 20 client.26666 trim_cache size 0 max 16384 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "3cd3809e-93b4-4713-8b43-2bbc4f5db15a", 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:23.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 msgr2=0x7f2e4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 0x7f2e4c0a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x5623871b2700 tx=0x7f2e5c009500 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 msgr2=0x7f2e64197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000e9c0 tx=0x7f2e5000edc0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 shutdown_connections 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f2e4c0a2800 0x7f2e4c0a4cc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e64107390 0x7f2e6419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e64104a60 0x7f2e64197fb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 --2- 172.21.15.123:0/3006243034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e64102130 0x7f2e64197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 >> 172.21.15.123:0/3006243034 conn(0x7f2e640fbb40 msgr2=0x7f2e64102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 shutdown_connections 2024-03-24T00:48:23.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.747+0000 7f2e6c400640 1 -- 172.21.15.123:0/3006243034 wait complete. 2024-03-24T00:48:23.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:48:23.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.963+0000 7fc79c200640 1 -- 172.21.15.123:0/1346229173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794104ad0 msgr2=0x7fc794104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.963+0000 7fc79c200640 1 --2- 172.21.15.123:0/1346229173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794104ad0 0x7fc794104f50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc780009900 tx=0x7fc78002bc40 comp rx=0 tx=0).stop 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.963+0000 7fc79c200640 1 -- 172.21.15.123:0/1346229173 shutdown_connections 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 --2- 172.21.15.123:0/1346229173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794105490 0x7fc79410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 --2- 172.21.15.123:0/1346229173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794104ad0 0x7fc794104f50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 --2- 172.21.15.123:0/1346229173 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 -- 172.21.15.123:0/1346229173 >> 172.21.15.123:0/1346229173 conn(0x7fc7940fee50 msgr2=0x7fc794101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 -- 172.21.15.123:0/1346229173 shutdown_connections 2024-03-24T00:48:23.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.964+0000 7fc79c200640 1 -- 172.21.15.123:0/1346229173 wait complete. 2024-03-24T00:48:23.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 Processor -- start 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 -- start start 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794104ad0 0x7fc79419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 0x7fc7941a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410f590 con 0x7fc794104ad0 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410f410 con 0x7fc794105490 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc79c200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410f710 con 0x7fc7941038e0 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.965+0000 7fc799800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc799800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:46916/0 (socket says 172.21.15.123:46916) 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc799800640 1 -- 172.21.15.123:0/1348564099 learned_addr learned my addr 172.21.15.123:0/1348564099 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 0x7fc7941a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc798e00640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794104ad0 0x7fc79419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 -- 172.21.15.123:0/1348564099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 msgr2=0x7fc794199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 -- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794104ad0 msgr2=0x7fc79419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794104ad0 0x7fc79419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 -- 172.21.15.123:0/1348564099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7941a17f0 con 0x7fc794105490 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc799800640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc79a200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 0x7fc7941a1180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc792000640 1 -- 172.21.15.123:0/1348564099 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00ce20 con 0x7fc794105490 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc792000640 1 -- 172.21.15.123:0/1348564099 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc78c005020 con 0x7fc794105490 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.966+0000 7fc792000640 1 -- 172.21.15.123:0/1348564099 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c010720 con 0x7fc794105490 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.967+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7941a1ae0 con 0x7fc794105490 2024-03-24T00:48:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.967+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7941a1fb0 con 0x7fc794105490 2024-03-24T00:48:23.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc792000640 1 -- 172.21.15.123:0/1348564099 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc78c005540 con 0x7fc794105490 2024-03-24T00:48:23.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 Processor -- start 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 -- start start 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f55e80 0x55d1f4f562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f5de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4ebb250 con 0x55d1f4f55e80 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4e63a30 con 0x55d1f4f56870 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.970+0000 7fc79f005740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4eb72f0 con 0x55d1f4f5b9e0 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc798e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f55e80 0x55d1f4f562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33962/0 (socket says 172.21.15.123:33962) 2024-03-24T00:48:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 -- 172.21.15.123:0/4218216816 learned_addr learned my addr 172.21.15.123:0/4218216816 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc798e00640 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f5de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 -- 172.21.15.123:0/4218216816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 msgr2=0x55d1f4f5de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f5de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 -- 172.21.15.123:0/4218216816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f55e80 msgr2=0x55d1f4f562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f55e80 0x55d1f4f562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 -- 172.21.15.123:0/4218216816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f4e4e690 con 0x55d1f4f56870 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc798e00640 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f5de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.971+0000 7fc799800640 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc788010580 tx=0x7fc78800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38f24b9872f45ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc786a00640 1 -- 172.21.15.123:0/4218216816 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788011760 con 0x55d1f4f56870 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc786a00640 1 -- 172.21.15.123:0/4218216816 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc788011de0 con 0x55d1f4f56870 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc786a00640 1 -- 172.21.15.123:0/4218216816 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78800e430 con 0x55d1f4f56870 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc792000640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 0x7fc77c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc792000640 1 -- 172.21.15.123:0/1348564099 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc78c0cf140 con 0x7fc794105490 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc798e00640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 0x7fc77c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc79f005740 1 -- 172.21.15.123:0/4218216816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 msgr2=0x55d1f4f5b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.972+0000 7fc79f005740 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc788010580 tx=0x7fc78800bf90 comp rx=0 tx=0).stop 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 -- 172.21.15.123:0/4218216816 shutdown_connections 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f5de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f56870 0x55d1f4f5b370 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 --2- 172.21.15.123:0/4218216816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f55e80 0x55d1f4f562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 -- 172.21.15.123:0/4218216816 >> 172.21.15.123:0/4218216816 conn(0x55d1f4f1e200 msgr2=0x55d1f4f517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 -- 172.21.15.123:0/4218216816 shutdown_connections 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 -- 172.21.15.123:0/4218216816 wait complete. 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc798e00640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 0x7fc77c0a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc78002c250 tx=0x7fc780000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.973+0000 7fc79f005740 1 -- start start 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 Processor -- start 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f56870 0x55d1f4f60310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f67350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4f5e990 con 0x55d1f4f5b9e0 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4e76e80 con 0x55d1f4f55e80 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4f5eb10 con 0x55d1f4f56870 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79f005740 10 client.0 _ll_register_callbacks cb 0x55d1f4e89130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc798e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f56870 0x55d1f4f60310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33976/0 (socket says 172.21.15.123:33976) 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc799800640 1 -- 172.21.15.123:0/113709126 learned_addr learned my addr 172.21.15.123:0/113709126 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.974+0000 7fc79a200640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f67350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 -- 172.21.15.123:0/113709126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f56870 msgr2=0x55d1f4f60310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f56870 0x55d1f4f60310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 -- 172.21.15.123:0/113709126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f5b9e0 msgr2=0x55d1f4f67350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f67350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 -- 172.21.15.123:0/113709126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1f4c997d0 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc799800640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc788010550 tx=0x7fc78801df70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6cdae6497da934 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc77b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78802b020 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc788005020 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc788011de0 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc788005220 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc79f005740 10 client.26678 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.975+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d1f4bcf3f0 con 0x55d1f4f55e80 2024-03-24T00:48:23.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc788011de0 con 0x55d1f4f55e80 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc788004750 con 0x55d1f4f55e80 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 10 client.26678 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 10 client.26678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 20 client.26678 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4f6b860 0x55d1f4f6dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 4 client.26678 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1f4ec8790 con 0x55d1f4f6b860 2024-03-24T00:48:23.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.976+0000 7fc798e00640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4f6b860 0x55d1f4f6dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.977+0000 7fc798e00640 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4f6b860 0x55d1f4f6dc50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5bcfebb5efeae5f server_cookie=86ca2e916c48405d in_seq=0 out_seq=0 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.977+0000 7fc77b400640 10 client.26678 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc77b400640 1 -- 172.21.15.123:0/113709126 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d1f4ec8790 con 0x55d1f4f6b860 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc77b400640 10 client.26678 handle_command_reply: tid=0 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 client.26678 shutdown 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 msgr2=0x55d1f4f5fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc788010550 tx=0x7fc78801df70 comp rx=0 tx=0).stop 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 shutdown_connections 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f67350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4f6b860 0x55d1f4f6dc50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f56870 0x55d1f4f60310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 --2- 172.21.15.123:0/113709126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f5fdd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:23.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 >> 172.21.15.123:0/113709126 conn(0x55d1f4f1e200 msgr2=0x55d1f4f50a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.978+0000 7fc77b400640 0 client.26678 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.979+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 shutdown_connections 2024-03-24T00:48:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.979+0000 7fc79f005740 1 -- 172.21.15.123:0/113709126 wait complete. 2024-03-24T00:48:23.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.979+0000 7fc79f005740 20 client.26678 trim_cache size 0 max 16384 2024-03-24T00:48:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.997+0000 7fc79f005740 1 Processor -- start 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.997+0000 7fc79f005740 1 -- start start 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f55e80 0x55d1f4f572e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f67310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941a8ce0 con 0x55d1f4f56870 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941ab2a0 con 0x55d1f4f5b9e0 2024-03-24T00:48:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79f005740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941a92e0 con 0x55d1f4f55e80 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc799800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f55e80 0x55d1f4f572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33984/0 (socket says 172.21.15.123:33984) 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 -- 172.21.15.123:0/2693188670 learned_addr learned my addr 172.21.15.123:0/2693188670 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc798e00640 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f67310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 -- 172.21.15.123:0/2693188670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f55e80 msgr2=0x55d1f4f572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f55e80 0x55d1f4f572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 -- 172.21.15.123:0/2693188670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 msgr2=0x55d1f4f67310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f67310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.998+0000 7fc79a200640 1 -- 172.21.15.123:0/2693188670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f4b8f0f0 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc79a200640 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c760 tx=0x7fc78c0054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31e51094a3e0e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc786000640 1 -- 172.21.15.123:0/2693188670 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0b9070 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc786000640 1 -- 172.21.15.123:0/2693188670 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc78c099d90 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc786000640 1 -- 172.21.15.123:0/2693188670 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0b4db0 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc79f005740 1 -- 172.21.15.123:0/2693188670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 msgr2=0x55d1f4f55140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:23.999+0000 7fc79f005740 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c760 tx=0x7fc78c0054b0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 -- 172.21.15.123:0/2693188670 shutdown_connections 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f55140 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f67310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 --2- 172.21.15.123:0/2693188670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f55e80 0x55d1f4f572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 -- 172.21.15.123:0/2693188670 >> 172.21.15.123:0/2693188670 conn(0x55d1f4f1e200 msgr2=0x55d1f4f54a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 -- 172.21.15.123:0/2693188670 shutdown_connections 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 -- 172.21.15.123:0/2693188670 wait complete. 2024-03-24T00:48:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 -- start start 2024-03-24T00:48:24.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.000+0000 7fc79f005740 1 Processor -- start 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f65730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f66740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941a9920 con 0x55d1f4f56870 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941aac40 con 0x55d1f4f55e80 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941a9f80 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79f005740 10 client.0 _ll_register_callbacks cb 0x7fc780009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc798e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f65730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc799800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33994/0 (socket says 172.21.15.123:33994) 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc799800640 1 -- 172.21.15.123:0/2992052861 learned_addr learned my addr 172.21.15.123:0/2992052861 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f66740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 -- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 msgr2=0x55d1f4f651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 -- 172.21.15.123:0/2992052861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 msgr2=0x55d1f4f65730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f65730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc79a200640 1 -- 172.21.15.123:0/2992052861 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1f4f66d30 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.001+0000 7fc799800640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79a200640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f66740 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc78c097ef0 tx=0x7fc78c0ccb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=883e0ad6217cfb27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c07e020 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc78c0051d0 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc78c099d90 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc78c0b9030 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 10 client.26686 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d1f4fc7860 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc78c099d90 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc78c09afe0 con 0x55d1f4f5b9e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 10 client.26686 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 10 client.26686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 20 client.26686 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.002+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4fc80e0 0x55d1f4fca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc79f005740 4 client.26686 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d1f4ec2ae0 con 0x55d1f4fc80e0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc799800640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4fc80e0 0x55d1f4fca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc799800640 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4fc80e0 0x55d1f4fca500 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21dcbb813dc1dc52 server_cookie=2b09a64ee0e52f48 in_seq=0 out_seq=0 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc77b400640 10 client.26686 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:24.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc77b400640 1 -- 172.21.15.123:0/2992052861 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55d1f4ec2ae0 con 0x55d1f4fc80e0 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.003+0000 7fc77b400640 10 client.26686 handle_command_reply: tid=0 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 client.26686 shutdown 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 msgr2=0x55d1f4f66740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f66740 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc78c097ef0 tx=0x7fc78c0ccb90 comp rx=0 tx=0).stop 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 shutdown_connections 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55d1f4fc80e0 0x55d1f4fca500 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d1f4f5b9e0 0x55d1f4f66740 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d1f4f56870 0x55d1f4f65730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc77b400640 0 client.26686 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 --2- 172.21.15.123:0/2992052861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d1f4f55e80 0x55d1f4f651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 >> 172.21.15.123:0/2992052861 conn(0x55d1f4f1e200 msgr2=0x55d1f4f54a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 shutdown_connections 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 1 -- 172.21.15.123:0/2992052861 wait complete. 2024-03-24T00:48:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.004+0000 7fc79f005740 20 client.26686 trim_cache size 0 max 16384 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "3cd3809e-93b4-4713-8b43-2bbc4f5db15a": { 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "3cd3809e-93b4-4713-8b43-2bbc4f5db15a", 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:48:24.010 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:24.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 msgr2=0x7fc77c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:24.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 0x7fc77c0a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc78002c250 tx=0x7fc780000bc0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 msgr2=0x7fc7941a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:24.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 0x7fc7941a1180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 shutdown_connections 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc794105490 0x7fc7941a1180 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fc77c0a2880 0x7fc77c0a4d40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794104ad0 0x7fc79419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 --2- 172.21.15.123:0/1348564099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7941038e0 0x7fc794199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 >> 172.21.15.123:0/1348564099 conn(0x7fc7940fee50 msgr2=0x7fc794107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 shutdown_connections 2024-03-24T00:48:24.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:24.005+0000 7fc79c200640 1 -- 172.21.15.123:0/1348564099 wait complete. 2024-03-24T00:48:24.025 INFO:tasks.cephfs.filesystem:scrub status for tag:3cd3809e-93b4-4713-8b43-2bbc4f5db15a - {'path': '/', 'tag': '3cd3809e-93b4-4713-8b43-2bbc4f5db15a', 'options': 'recursive,force'} 2024-03-24T00:48:24.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:24 smithi177 ceph-mon[36034]: pgmap v700: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:24.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:24 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:48:24.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:24 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:24.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:24 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:48:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:24 smithi204 ceph-mon[36345]: pgmap v700: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:24 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:48:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:24 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:24 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:48:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:24 smithi123 ceph-mon[31487]: pgmap v700: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:24 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:48:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:24 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:24.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:24 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:48:25.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:25 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:25.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:25 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:25.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:25 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:26.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:26 smithi177 ceph-mon[36034]: pgmap v701: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-24T00:48:26.722 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:26 smithi204 ceph-mon[36345]: pgmap v701: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-24T00:48:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:26 smithi123 ceph-mon[31487]: pgmap v701: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 29 op/s 2024-03-24T00:48:27.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:27 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:48:27.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:27 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:48:27.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:27 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:48:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:28 smithi204 ceph-mon[36345]: pgmap v702: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 33 op/s 2024-03-24T00:48:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:28 smithi123 ceph-mon[31487]: pgmap v702: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 33 op/s 2024-03-24T00:48:28.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:28 smithi177 ceph-mon[36034]: pgmap v702: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 33 op/s 2024-03-24T00:48:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:30 smithi204 ceph-mon[36345]: pgmap v703: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 KiB/s wr, 37 op/s 2024-03-24T00:48:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:30 smithi123 ceph-mon[31487]: pgmap v703: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 KiB/s wr, 37 op/s 2024-03-24T00:48:30.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:30 smithi177 ceph-mon[36034]: pgmap v703: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 KiB/s wr, 37 op/s 2024-03-24T00:48:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:32 smithi204 ceph-mon[36345]: pgmap v704: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-03-24T00:48:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:32 smithi123 ceph-mon[31487]: pgmap v704: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-03-24T00:48:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:32 smithi177 ceph-mon[36034]: pgmap v704: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-03-24T00:48:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:34 smithi204 ceph-mon[36345]: pgmap v705: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.3 KiB/s wr, 35 op/s 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 -- 172.21.15.123:0/463336953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 msgr2=0x7fef90103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 --2- 172.21.15.123:0/463336953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90103d20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fef80009860 tx=0x7fef8002bb70 comp rx=0 tx=0).stop 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 -- 172.21.15.123:0/463336953 shutdown_connections 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 --2- 172.21.15.123:0/463336953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 0x7fef9010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 --2- 172.21.15.123:0/463336953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 0x7fef90104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 --2- 172.21.15.123:0/463336953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90103d20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 -- 172.21.15.123:0/463336953 >> 172.21.15.123:0/463336953 conn(0x7fef900fee90 msgr2=0x7fef901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 -- 172.21.15.123:0/463336953 shutdown_connections 2024-03-24T00:48:34.826 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.825+0000 7fef97000640 1 -- 172.21.15.123:0/463336953 wait complete. 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 Processor -- start 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 -- start start 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 0x7fef9019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 0x7fef901a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9010f550 con 0x7fef90104af0 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9010f3d0 con 0x7fef90103900 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef97000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9010f6d0 con 0x7fef901054b0 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.826+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44506/0 (socket says 172.21.15.123:44506) 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 -- 172.21.15.123:0/1934506200 learned_addr learned my addr 172.21.15.123:0/1934506200 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef87400640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 0x7fef9019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef95000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 0x7fef901a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 -- 172.21.15.123:0/1934506200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 msgr2=0x7fef901a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 0x7fef901a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 -- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 msgr2=0x7fef9019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 0x7fef9019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 -- 172.21.15.123:0/1934506200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef901a1750 con 0x7fef90103900 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fef800349e0 tx=0x7fef80032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8d600640 1 -- 172.21.15.123:0/1934506200 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80041020 con 0x7fef90103900 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8d600640 1 -- 172.21.15.123:0/1934506200 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef8002c9f0 con 0x7fef90103900 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.827+0000 7fef8d600640 1 -- 172.21.15.123:0/1934506200 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80039040 con 0x7fef90103900 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.828+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef901a19e0 con 0x7fef90103900 2024-03-24T00:48:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.828+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef901a1e80 con 0x7fef90103900 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 Processor -- start 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef8d600640 1 -- 172.21.15.123:0/1934506200 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fef80044020 con 0x7fef90103900 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 -- start start 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa471a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa471f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa47221d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa46dd3b0 con 0x557fa471fd40 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa467f830 con 0x557fa471abd0 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.832+0000 7fef99d64740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa46285d0 con 0x557fa471a1e0 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef95000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa471a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa471f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef95000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa471a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:53598/0 (socket says 172.21.15.123:53598) 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef95000640 1 -- 172.21.15.123:0/1239621818 learned_addr learned my addr 172.21.15.123:0/1239621818 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef87400640 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa47221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 -- 172.21.15.123:0/1239621818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 msgr2=0x557fa471a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa471a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 -- 172.21.15.123:0/1239621818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 msgr2=0x557fa47221d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa47221d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 -- 172.21.15.123:0/1239621818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fa46402c0 con 0x557fa471abd0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8d600640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 0x7fef740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.833+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa471f6d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fef800edce0 tx=0x7fef800f8320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5079b219897b2a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef85600640 1 -- 172.21.15.123:0/1239621818 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800b8e80 con 0x557fa471abd0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef95000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 0x7fef740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef85600640 1 -- 172.21.15.123:0/1239621818 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef800fd030 con 0x557fa471abd0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef85600640 1 -- 172.21.15.123:0/1239621818 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800ccae0 con 0x557fa471abd0 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef8d600640 1 -- 172.21.15.123:0/1934506200 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fef800f0290 con 0x7fef90103900 2024-03-24T00:48:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef99d64740 1 -- 172.21.15.123:0/1239621818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 msgr2=0x557fa471f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef99d64740 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa471f6d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fef800edce0 tx=0x7fef800f8320 comp rx=0 tx=0).stop 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.834+0000 7fef95000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 0x7fef740a4de0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x557fa471b430 tx=0x7fef88009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 -- 172.21.15.123:0/1239621818 shutdown_connections 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa47221d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa471f6d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 --2- 172.21.15.123:0/1239621818 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa471a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 -- 172.21.15.123:0/1239621818 >> 172.21.15.123:0/1239621818 conn(0x557fa4715750 msgr2=0x557fa4715b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 -- 172.21.15.123:0/1239621818 shutdown_connections 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 -- 172.21.15.123:0/1239621818 wait complete. 2024-03-24T00:48:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 -- start start 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.835+0000 7fef99d64740 1 Processor -- start 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa4724080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa472b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa463ba20 con 0x557fa471fd40 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa467b8d0 con 0x557fa471abd0 2024-03-24T00:48:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fa4722ab0 con 0x557fa471a1e0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef99d64740 10 client.0 _ll_register_callbacks cb 0x557fa46781c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa4724080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef95000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa472b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa4724080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:53612/0 (socket says 172.21.15.123:53612) 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44522/0 (socket says 172.21.15.123:44522) 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef8fe00640 1 -- 172.21.15.123:0/1361138754 learned_addr learned my addr 172.21.15.123:0/1361138754 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 -- 172.21.15.123:0/1361138754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 msgr2=0x557fa4724080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa4724080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 -- 172.21.15.123:0/1361138754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 msgr2=0x557fa472b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa472b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef87400640 1 -- 172.21.15.123:0/1361138754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557fa4394210 con 0x557fa471abd0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.836+0000 7fef95000640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa472b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef87400640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fef9019b040 tx=0x7fef78012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ae61e47882aaf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef7e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef78004020 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fef780041c0 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef800fd030 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef780043c0 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef99d64740 10 client.26696 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557fa4460620 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.837+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef800fd030 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fef78003ba0 con 0x557fa471abd0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 10 client.26696 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 10 client.26696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 20 client.26696 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa472fed0 0x557fa4732310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 4 client.26696 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557fa468cd70 con 0x557fa472fed0 2024-03-24T00:48:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.838+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa472fed0 0x557fa4732310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa472fed0 0x557fa4732310 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3942969fa08a028 server_cookie=3aa0b87430d9ae9d in_seq=0 out_seq=0 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef7e000640 10 client.26696 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef7e000640 1 -- 172.21.15.123:0/1361138754 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557fa468cd70 con 0x557fa472fed0 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef7e000640 10 client.26696 handle_command_reply: tid=0 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef99d64740 1 client.26696 shutdown 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 msgr2=0x557fa47245c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.839+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fef9019b040 tx=0x7fef78012b60 comp rx=0 tx=0).stop 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 shutdown_connections 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471fd40 0x557fa472b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471abd0 0x557fa47245c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa472fed0 0x557fa4732310 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 --2- 172.21.15.123:0/1361138754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471a1e0 0x557fa4724080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef7e000640 0 client.26696 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 >> 172.21.15.123:0/1361138754 conn(0x557fa4715750 msgr2=0x557fa4714950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 shutdown_connections 2024-03-24T00:48:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 1 -- 172.21.15.123:0/1361138754 wait complete. 2024-03-24T00:48:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.840+0000 7fef99d64740 20 client.26696 trim_cache size 0 max 16384 2024-03-24T00:48:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 Processor -- start 2024-03-24T00:48:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 -- start start 2024-03-24T00:48:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 0x557fa472b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:34 smithi123 ceph-mon[31487]: pgmap v705: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.3 KiB/s wr, 35 op/s 2024-03-24T00:48:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471fd40 0x557fa47189b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901a8c40 con 0x557fa471a1e0 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901ab200 con 0x557fa471fd40 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef99d64740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901a9240 con 0x557fa471abd0 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef95000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471fd40 0x557fa47189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58878/0 (socket says 172.21.15.123:58878) 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef87400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 0x557fa472b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.855+0000 7fef8fe00640 1 -- 172.21.15.123:0/1528393401 learned_addr learned my addr 172.21.15.123:0/1528393401 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 -- 172.21.15.123:0/1528393401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 msgr2=0x557fa472b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 0x557fa472b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 -- 172.21.15.123:0/1528393401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471fd40 msgr2=0x557fa47189b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471fd40 0x557fa47189b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 -- 172.21.15.123:0/1528393401 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fa4725070 con 0x557fa471a1e0 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef87400640 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 0x557fa472b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef8fe00640 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fef8002c180 tx=0x7fef8002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6326895164e263 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef84c00640 1 -- 172.21.15.123:0/1528393401 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800a7040 con 0x557fa471a1e0 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef84c00640 1 -- 172.21.15.123:0/1528393401 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef800fd040 con 0x557fa471a1e0 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.856+0000 7fef84c00640 1 -- 172.21.15.123:0/1528393401 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800cd220 con 0x557fa471a1e0 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.857+0000 7fef99d64740 1 -- 172.21.15.123:0/1528393401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 msgr2=0x557fa4722de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.857+0000 7fef99d64740 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fef8002c180 tx=0x7fef8002bb70 comp rx=0 tx=0).stop 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.857+0000 7fef99d64740 1 -- 172.21.15.123:0/1528393401 shutdown_connections 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.858+0000 7fef99d64740 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471fd40 0x557fa47189b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.858+0000 7fef99d64740 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471abd0 0x557fa472b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.858+0000 7fef99d64740 1 --2- 172.21.15.123:0/1528393401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471a1e0 0x557fa4722de0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.858+0000 7fef99d64740 1 -- 172.21.15.123:0/1528393401 >> 172.21.15.123:0/1528393401 conn(0x557fa4715750 msgr2=0x557fa471be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.858+0000 7fef99d64740 1 -- 172.21.15.123:0/1528393401 shutdown_connections 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.859+0000 7fef99d64740 1 -- 172.21.15.123:0/1528393401 wait complete. 2024-03-24T00:48:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.859+0000 7fef99d64740 1 -- start start 2024-03-24T00:48:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.859+0000 7fef99d64740 1 Processor -- start 2024-03-24T00:48:34.862 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 0x557fa47297d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471fd40 0x557fa472a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901a9880 con 0x557fa471abd0 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901aaba0 con 0x557fa471a1e0 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901a9ee0 con 0x557fa471fd40 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.861+0000 7fef99d64740 10 client.0 _ll_register_callbacks cb 0x7fef78009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.862+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.862+0000 7fef8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:44538/0 (socket says 172.21.15.123:44538) 2024-03-24T00:48:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.862+0000 7fef8fe00640 1 -- 172.21.15.123:0/3585730097 learned_addr learned my addr 172.21.15.123:0/3585730097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.863+0000 7fef87400640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 0x557fa47297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.863+0000 7fef95000640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471fd40 0x557fa472a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 -- 172.21.15.123:0/3585730097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471fd40 msgr2=0x557fa472a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471fd40 0x557fa472a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 -- 172.21.15.123:0/3585730097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 msgr2=0x557fa47297d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 0x557fa47297d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 -- 172.21.15.123:0/3585730097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557fa472add0 con 0x557fa471a1e0 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef87400640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 0x557fa47297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.864+0000 7fef8fe00640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fef8002cbc0 tx=0x7fef800c5380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46631abf0f56c702 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.865+0000 7fef7e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.865+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800a7040 con 0x557fa471a1e0 2024-03-24T00:48:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.865+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fef800baee0 con 0x557fa471a1e0 2024-03-24T00:48:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.865+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef800fd040 con 0x557fa471a1e0 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.865+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef800cdd30 con 0x557fa471a1e0 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 10 client.26702 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557fa47944f0 con 0x557fa471a1e0 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef800fd040 con 0x557fa471a1e0 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fef8003c030 con 0x557fa471a1e0 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 10 client.26702 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 10 client.26702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 20 client.26702 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa4794c40 0x557fa4797060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 4 client.26702 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:48:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.866+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557fa4710a80 con 0x557fa4794c40 2024-03-24T00:48:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.867+0000 7fef87400640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa4794c40 0x557fa4797060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.867+0000 7fef87400640 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa4794c40 0x557fa4797060 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec0c49148500a3a server_cookie=c67678de7e1c27e1 in_seq=0 out_seq=0 2024-03-24T00:48:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:34.867+0000 7fef7e000640 10 client.26702 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:34.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:34 smithi177 ceph-mon[36034]: pgmap v705: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.3 KiB/s wr, 35 op/s 2024-03-24T00:48:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:36 smithi204 ceph-mon[36345]: pgmap v706: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 37 op/s 2024-03-24T00:48:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:36 smithi123 ceph-mon[31487]: pgmap v706: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 37 op/s 2024-03-24T00:48:36.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:36 smithi177 ceph-mon[36034]: pgmap v706: 97 pgs: 97 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 37 op/s 2024-03-24T00:48:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:37 smithi204 ceph-mon[36345]: pgmap v707: 97 pgs: 97 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.0 KiB/s wr, 25 op/s 2024-03-24T00:48:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:37 smithi123 ceph-mon[31487]: pgmap v707: 97 pgs: 97 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.0 KiB/s wr, 25 op/s 2024-03-24T00:48:37.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:37 smithi177 ceph-mon[36034]: pgmap v707: 97 pgs: 97 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.0 KiB/s wr, 25 op/s 2024-03-24T00:48:40.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:40 smithi177 ceph-mon[36034]: pgmap v708: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-24T00:48:40.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:40 smithi204 ceph-mon[36345]: pgmap v708: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-24T00:48:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:40 smithi123 ceph-mon[31487]: pgmap v708: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-24T00:48:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:42 smithi204 ceph-mon[36345]: pgmap v709: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 7.6 KiB/s wr, 13 op/s 2024-03-24T00:48:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:42 smithi123 ceph-mon[31487]: pgmap v709: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 7.6 KiB/s wr, 13 op/s 2024-03-24T00:48:42.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:42 smithi177 ceph-mon[36034]: pgmap v709: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 7.6 KiB/s wr, 13 op/s 2024-03-24T00:48:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.485+0000 7fef7e000640 1 -- 172.21.15.123:0/3585730097 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557fa4710a80 con 0x557fa4794c40 2024-03-24T00:48:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.485+0000 7fef7e000640 10 client.26702 handle_command_reply: tid=0 2024-03-24T00:48:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.485+0000 7fef99d64740 1 client.26702 shutdown 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.485+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 msgr2=0x557fa4729290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.485+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fef8002cbc0 tx=0x7fef800c5380 comp rx=0 tx=0).stop 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 shutdown_connections 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557fa4794c40 0x557fa4797060 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557fa471fd40 0x557fa472a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557fa471abd0 0x557fa47297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 --2- 172.21.15.123:0/3585730097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557fa471a1e0 0x557fa4729290 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 >> 172.21.15.123:0/3585730097 conn(0x557fa4715750 msgr2=0x557fa47189b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef7e000640 0 client.26702 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 shutdown_connections 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 1 -- 172.21.15.123:0/3585730097 wait complete. 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.486+0000 7fef99d64740 20 client.26702 trim_cache size 0 max 16384 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:48:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.487+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 msgr2=0x7fef740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.487+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 0x7fef740a4de0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x557fa471b430 tx=0x7fef88009500 comp rx=0 tx=0).stop 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.487+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 msgr2=0x7fef90199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.487+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fef800349e0 tx=0x7fef80032430 comp rx=0 tx=0).stop 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 shutdown_connections 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef901054b0 0x7fef901a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fef740a2920 0x7fef740a4de0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef90104af0 0x7fef9019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 --2- 172.21.15.123:0/1934506200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef90103900 0x7fef90199b30 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 >> 172.21.15.123:0/1934506200 conn(0x7fef900fee90 msgr2=0x7fef90107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 shutdown_connections 2024-03-24T00:48:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:44.488+0000 7fef97000640 1 -- 172.21.15.123:0/1934506200 wait complete. 2024-03-24T00:48:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:44 smithi204 ceph-mon[36345]: pgmap v710: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-03-24T00:48:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:44 smithi123 ceph-mon[31487]: pgmap v710: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-03-24T00:48:44.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:44 smithi177 ceph-mon[36034]: pgmap v710: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-03-24T00:48:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:46 smithi204 ceph-mon[36345]: pgmap v711: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-24T00:48:46.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:46 smithi123 ceph-mon[31487]: pgmap v711: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-24T00:48:46.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:46 smithi177 ceph-mon[36034]: pgmap v711: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-03-24T00:48:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:48 smithi204 ceph-mon[36345]: pgmap v712: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-24T00:48:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:48 smithi123 ceph-mon[31487]: pgmap v712: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-24T00:48:48.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:48 smithi177 ceph-mon[36034]: pgmap v712: 97 pgs: 97 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-03-24T00:48:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:50 smithi204 ceph-mon[36345]: pgmap v713: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:50.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:50 smithi123 ceph-mon[31487]: pgmap v713: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:50.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:50 smithi177 ceph-mon[36034]: pgmap v713: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-24T00:48:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:52 smithi204 ceph-mon[36345]: pgmap v714: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-24T00:48:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:52 smithi123 ceph-mon[31487]: pgmap v714: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-24T00:48:52.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:52 smithi177 ceph-mon[36034]: pgmap v714: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-03-24T00:48:54.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:48:54.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 -- 172.21.15.123:0/3719857815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc341038c0 msgr2=0x7fdc34103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 --2- 172.21.15.123:0/3719857815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc341038c0 0x7fdc34103ce0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009900 tx=0x7fdc2402bc50 comp rx=0 tx=0).stop 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 -- 172.21.15.123:0/3719857815 shutdown_connections 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 --2- 172.21.15.123:0/3719857815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc34105470 0x7fdc3410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 --2- 172.21.15.123:0/3719857815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34104ab0 0x7fdc34104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 --2- 172.21.15.123:0/3719857815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc341038c0 0x7fdc34103ce0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 -- 172.21.15.123:0/3719857815 >> 172.21.15.123:0/3719857815 conn(0x7fdc340fee50 msgr2=0x7fdc34101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 -- 172.21.15.123:0/3719857815 shutdown_connections 2024-03-24T00:48:54.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.235+0000 7fdc39000640 1 -- 172.21.15.123:0/3719857815 wait complete. 2024-03-24T00:48:54.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.236+0000 7fdc39000640 1 Processor -- start 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 -- start start 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc341038c0 0x7fdc34199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 0x7fdc341a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3410f580 con 0x7fdc341038c0 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3410f400 con 0x7fdc34104ab0 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc39000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3410f700 con 0x7fdc34105470 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc341038c0 0x7fdc34199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc31400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc31400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46526/0 (socket says 172.21.15.123:46526) 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc31400640 1 -- 172.21.15.123:0/3341423010 learned_addr learned my addr 172.21.15.123:0/3341423010 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.237+0000 7fdc32800640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 0x7fdc341a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 -- 172.21.15.123:0/3341423010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 msgr2=0x7fdc341a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 0x7fdc341a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 -- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc341038c0 msgr2=0x7fdc34199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc341038c0 0x7fdc34199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 -- 172.21.15.123:0/3341423010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc341a1850 con 0x7fdc34104ab0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc32800640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 0x7fdc341a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc31400640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000e9f0 tx=0x7fdc2000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc2aa00640 1 -- 172.21.15.123:0/3341423010 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000cd80 con 0x7fdc34104ab0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc2aa00640 1 -- 172.21.15.123:0/3341423010 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc20005500 con 0x7fdc34104ab0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc341a1b40 con 0x7fdc34104ab0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc2aa00640 1 -- 172.21.15.123:0/3341423010 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20010690 con 0x7fdc34104ab0 2024-03-24T00:48:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.238+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc341a2060 con 0x7fdc34104ab0 2024-03-24T00:48:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 Processor -- start 2024-03-24T00:48:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 -- start start 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875614370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x564875616e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875574250 con 0x5648756149e0 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487551ca30 con 0x56487560ee80 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648755702f0 con 0x56487560f870 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc32800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc2aa00640 1 -- 172.21.15.123:0/3341423010 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdc200046c0 con 0x7fdc34104ab0 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875614370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.243+0000 7fdc32800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46528/0 (socket says 172.21.15.123:46528) 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 -- 172.21.15.123:0/3828710945 learned_addr learned my addr 172.21.15.123:0/3828710945 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc31400640 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x564875616e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 -- 172.21.15.123:0/3828710945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 msgr2=0x564875614370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875614370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 -- 172.21.15.123:0/3828710945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 msgr2=0x564875616e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x564875616e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 -- 172.21.15.123:0/3828710945 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564875507690 con 0x56487560ee80 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc31400640 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x564875616e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.244+0000 7fdc32800640 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0117e0 tx=0x7fdc2c011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf95be2ba19ca07e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc1f400640 1 -- 172.21.15.123:0/3828710945 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c011310 con 0x56487560ee80 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc1f400640 1 -- 172.21.15.123:0/3828710945 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c003e50 con 0x56487560ee80 2024-03-24T00:48:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc1f400640 1 -- 172.21.15.123:0/3828710945 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c004620 con 0x56487560ee80 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc2aa00640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 0x7fdc180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc2aa00640 1 -- 172.21.15.123:0/3341423010 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdc200ceec0 con 0x7fdc34104ab0 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3828710945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 msgr2=0x56487560f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0117e0 tx=0x7fdc2c011ca0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 0x7fdc180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3828710945 shutdown_connections 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x564875616e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875614370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3828710945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x56487560f2a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.245+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3828710945 >> 172.21.15.123:0/3828710945 conn(0x5648755d7200 msgr2=0x56487560a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.246+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3828710945 shutdown_connections 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.246+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3828710945 wait complete. 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.246+0000 7fdc3bc30740 1 -- start start 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.246+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 0x7fdc180a4d40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402c190 tx=0x7fdc24000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.246+0000 7fdc3bc30740 1 Processor -- start 2024-03-24T00:48:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875619390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x5648756203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875617990 con 0x5648756149e0 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487552fe80 con 0x56487560ee80 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875617b10 con 0x56487560f870 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc3bc30740 10 client.0 _ll_register_callbacks cb 0x564875542130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46544/0 (socket says 172.21.15.123:46544) 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 -- 172.21.15.123:0/3982068868 learned_addr learned my addr 172.21.15.123:0/3982068868 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31400640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875619390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc32800640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x5648756203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 -- 172.21.15.123:0/3982068868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 msgr2=0x564875619390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875619390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 -- 172.21.15.123:0/3982068868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 msgr2=0x5648756203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x5648756203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc31e00640 1 -- 172.21.15.123:0/3982068868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648753527d0 con 0x56487560ee80 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.247+0000 7fdc32800640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x5648756203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdc34104910 tx=0x7fdc24009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74d38c6a06359cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc13e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc240050b0 con 0x56487560ee80 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdc24005250 con 0x56487560ee80 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc2c003e50 con 0x56487560ee80 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc24005450 con 0x56487560ee80 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc3bc30740 10 client.26714 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.248+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5648752883f0 con 0x56487560ee80 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c003e50 con 0x56487560ee80 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdc2405c020 con 0x56487560ee80 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 10 client.26714 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 10 client.26714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 20 client.26714 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875624b00 0x564875626f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 4 client.26714 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564875581790 con 0x564875624b00 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.249+0000 7fdc31400640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875624b00 0x564875626f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc13e00640 10 client.26714 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc31400640 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875624b00 0x564875626f10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b76186b46a2216f8 server_cookie=643b76e0e706ed73 in_seq=0 out_seq=0 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc13e00640 1 -- 172.21.15.123:0/3982068868 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564875581790 con 0x564875624b00 2024-03-24T00:48:54.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc13e00640 10 client.26714 handle_command_reply: tid=0 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc3bc30740 1 client.26714 shutdown 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 msgr2=0x564875618e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.250+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdc34104910 tx=0x7fdc24009260 comp rx=0 tx=0).stop 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 shutdown_connections 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x5648756203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875624b00 0x564875626f10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875619390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3982068868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560ee80 0x564875618e50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 >> 172.21.15.123:0/3982068868 conn(0x5648755d7200 msgr2=0x564875609a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc13e00640 0 client.26714 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 shutdown_connections 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3982068868 wait complete. 2024-03-24T00:48:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.251+0000 7fdc3bc30740 20 client.26714 trim_cache size 0 max 16384 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.268+0000 7fdc3bc30740 1 Processor -- start 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 -- start start 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56487560ee80 0x564875619170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875620360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341a8cc0 con 0x56487560ee80 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341ab280 con 0x5648756149e0 2024-03-24T00:48:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341a92c0 con 0x56487560f870 2024-03-24T00:48:54.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc32800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56487560ee80 0x564875619170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc32800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46556/0 (socket says 172.21.15.123:46556) 2024-03-24T00:48:54.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc32800640 1 -- 172.21.15.123:0/998769182 learned_addr learned my addr 172.21.15.123:0/998769182 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.269+0000 7fdc31400640 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875620360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 -- 172.21.15.123:0/998769182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 msgr2=0x564875620360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875620360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 -- 172.21.15.123:0/998769182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56487560ee80 msgr2=0x564875619170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56487560ee80 0x564875619170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 -- 172.21.15.123:0/998769182 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648752480f0 con 0x5648756149e0 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc32800640 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005c40 tx=0x7fdc2c011280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7390f680160c7fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc1ea00640 1 -- 172.21.15.123:0/998769182 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c02a070 con 0x5648756149e0 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc1ea00640 1 -- 172.21.15.123:0/998769182 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c00fe80 con 0x5648756149e0 2024-03-24T00:48:54.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.270+0000 7fdc1ea00640 1 -- 172.21.15.123:0/998769182 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0054c0 con 0x5648756149e0 2024-03-24T00:48:54.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- 172.21.15.123:0/998769182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 msgr2=0x564875610270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005c40 tx=0x7fdc2c011280 comp rx=0 tx=0).stop 2024-03-24T00:48:54.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- 172.21.15.123:0/998769182 shutdown_connections 2024-03-24T00:48:54.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648756149e0 0x564875610270 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560f870 0x564875620360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/998769182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56487560ee80 0x564875619170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- 172.21.15.123:0/998769182 >> 172.21.15.123:0/998769182 conn(0x5648755d7200 msgr2=0x564875618a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- 172.21.15.123:0/998769182 shutdown_connections 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- 172.21.15.123:0/998769182 wait complete. 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.271+0000 7fdc3bc30740 1 -- start start 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 Processor -- start 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560ee80 0x56487561d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x56487561e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341a9900 con 0x5648756149e0 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341aac20 con 0x56487560f870 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341a9f60 con 0x56487560ee80 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc3bc30740 10 client.0 _ll_register_callbacks cb 0x7fdc24009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:54.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc31400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc31400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46570/0 (socket says 172.21.15.123:46570) 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc31e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560ee80 0x56487561d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc31400640 1 -- 172.21.15.123:0/3786296889 learned_addr learned my addr 172.21.15.123:0/3786296889 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.272+0000 7fdc32800640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x56487561e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 -- 172.21.15.123:0/3786296889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560ee80 msgr2=0x56487561d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560ee80 0x56487561d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 -- 172.21.15.123:0/3786296889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 msgr2=0x56487561e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x56487561e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 -- 172.21.15.123:0/3786296889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56487561ef70 con 0x56487560f870 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc32800640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x56487561e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc31400640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000c6a0 tx=0x7fdc20086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4943e788ed58558 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc13e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:54.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2009b810 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdc20097df0 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56487557bae0 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc2009f960 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc3bc30740 10 client.26720 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.273+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564875620350 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56487557bae0 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdc20097040 con 0x56487560f870 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 10 client.26720 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:54.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 10 client.26720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 20 client.26720 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875680750 0x564875682b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 4 client.26720 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5648754f93a0 con 0x564875680750 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875680750 0x564875682b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc31e00640 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875680750 0x564875682b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90fe4b6c0a1c717d server_cookie=ab0829e01dcc7760 in_seq=0 out_seq=0 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.274+0000 7fdc13e00640 10 client.26720 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc13e00640 1 -- 172.21.15.123:0/3786296889 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5648754f93a0 con 0x564875680750 2024-03-24T00:48:54.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc13e00640 10 client.26720 handle_command_reply: tid=0 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 client.26720 shutdown 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 msgr2=0x56487561d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000c6a0 tx=0x7fdc20086040 comp rx=0 tx=0).stop 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 shutdown_connections 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648756149e0 0x56487561e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56487560f870 0x56487561d970 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564875680750 0x564875682b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 --2- 172.21.15.123:0/3786296889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56487560ee80 0x56487561d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 >> 172.21.15.123:0/3786296889 conn(0x5648755d7200 msgr2=0x56487560da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc13e00640 0 client.26720 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 shutdown_connections 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 1 -- 172.21.15.123:0/3786296889 wait complete. 2024-03-24T00:48:54.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.275+0000 7fdc3bc30740 20 client.26720 trim_cache size 0 max 16384 2024-03-24T00:48:54.283 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:54.283 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:48:54.283 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:48:54.283 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:54.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 msgr2=0x7fdc180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 0x7fdc180a4d40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402c190 tx=0x7fdc24000bc0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 msgr2=0x7fdc3419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000e9f0 tx=0x7fdc2000edf0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 shutdown_connections 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc34105470 0x7fdc341a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc34104ab0 0x7fdc3419a180 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fdc180a2880 0x7fdc180a4d40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 --2- 172.21.15.123:0/3341423010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc341038c0 0x7fdc34199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 >> 172.21.15.123:0/3341423010 conn(0x7fdc340fee50 msgr2=0x7fdc34107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 shutdown_connections 2024-03-24T00:48:54.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.276+0000 7fdc39000640 1 -- 172.21.15.123:0/3341423010 wait complete. 2024-03-24T00:48:54.293 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:48:54.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 -- 172.21.15.123:0/78756662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 msgr2=0x7fbc1010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/78756662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc1010ff40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009ca0 tx=0x7fbc0802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 -- 172.21.15.123:0/78756662 shutdown_connections 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/78756662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc1010ff40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/78756662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc10109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/78756662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 0x7fbc10107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 -- 172.21.15.123:0/78756662 >> 172.21.15.123:0/78756662 conn(0x7fbc10076ab0 msgr2=0x7fbc10078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 -- 172.21.15.123:0/78756662 shutdown_connections 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.471+0000 7fbc0fe00640 1 -- 172.21.15.123:0/78756662 wait complete. 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 Processor -- start 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 -- start start 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 0x7fbc1019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc101a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10113780 con 0x7fbc10107b00 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10113600 con 0x7fbc101096b0 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.472+0000 7fbc0fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10113900 con 0x7fbc10108cf0 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 0x7fbc1019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56430/0 (socket says 172.21.15.123:56430) 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0e000640 1 -- 172.21.15.123:0/2550112615 learned_addr learned my addr 172.21.15.123:0/2550112615 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc101a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 -- 172.21.15.123:0/2550112615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 msgr2=0x7fbc1019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 -- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 msgr2=0x7fbc1019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 0x7fbc1019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 -- 172.21.15.123:0/2550112615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc101a5980 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0e000640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.473+0000 7fbc0f400640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc101a5310 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbc08037040 tx=0x7fbc08032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.474+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08041020 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.474+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc08034cd0 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.474+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc101a5c10 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.474+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08039040 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.474+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc101a6120 con 0x7fbc101096b0 2024-03-24T00:48:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.475+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1010edc0 con 0x7fbc101096b0 2024-03-24T00:48:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.477+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbc08044020 con 0x7fbc101096b0 2024-03-24T00:48:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.478+0000 7fbc07400640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 0x7fbbf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.478+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbc080f0b10 con 0x7fbc101096b0 2024-03-24T00:48:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.478+0000 7fbc0ea00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 0x7fbbf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.479+0000 7fbc0ea00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 0x7fbbf40a4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbc10108b50 tx=0x7fbbfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.485+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbc0804b020 con 0x7fbc101096b0 2024-03-24T00:48:54.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:54 smithi123 ceph-mon[31487]: pgmap v715: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-03-24T00:48:54.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.740+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc1002d070 con 0x7fbc101096b0 2024-03-24T00:48:54.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.741+0000 7fbc07400640 1 -- 172.21.15.123:0/2550112615 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fbc080bdbf0 con 0x7fbc101096b0 2024-03-24T00:48:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:48:54.742 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:48:54.742 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 msgr2=0x7fbbf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 0x7fbbf40a4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbc10108b50 tx=0x7fbbfc005f90 comp rx=0 tx=0).stop 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 msgr2=0x7fbc101a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc101a5310 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbc08037040 tx=0x7fbc08032ea0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 shutdown_connections 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc101096b0 0x7fbc101a5310 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc10108cf0 0x7fbc1019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fbbf40a2880 0x7fbbf40a4d40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 --2- 172.21.15.123:0/2550112615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc10107b00 0x7fbc1019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 >> 172.21.15.123:0/2550112615 conn(0x7fbc10076ab0 msgr2=0x7fbc1010bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 shutdown_connections 2024-03-24T00:48:54.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.744+0000 7fbc0fe00640 1 -- 172.21.15.123:0/2550112615 wait complete. 2024-03-24T00:48:54.763 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:48:54.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:48:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:54 smithi204 ceph-mon[36345]: pgmap v715: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-03-24T00:48:54.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 -- 172.21.15.123:0/851286117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434105490 msgr2=0x7f943410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 --2- 172.21.15.123:0/851286117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434105490 0x7f943410bd20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f942c009ca0 tx=0x7f942c02c100 comp rx=0 tx=0).stop 2024-03-24T00:48:54.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 -- 172.21.15.123:0/851286117 shutdown_connections 2024-03-24T00:48:54.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 --2- 172.21.15.123:0/851286117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434105490 0x7f943410bd20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 --2- 172.21.15.123:0/851286117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434104ad0 0x7f9434104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 --2- 172.21.15.123:0/851286117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 -- 172.21.15.123:0/851286117 >> 172.21.15.123:0/851286117 conn(0x7f94340fee50 msgr2=0x7f9434101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 -- 172.21.15.123:0/851286117 shutdown_connections 2024-03-24T00:48:54.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.958+0000 7f943bc00640 1 -- 172.21.15.123:0/851286117 wait complete. 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 Processor -- start 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 -- start start 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434104ad0 0x7f943419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 0x7f94341a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943410f560 con 0x7f9434105490 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943410f3e0 con 0x7f9434104ad0 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.959+0000 7f943bc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943410f6e0 con 0x7f94341038e0 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56444/0 (socket says 172.21.15.123:56444) 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 -- 172.21.15.123:0/2442696598 learned_addr learned my addr 172.21.15.123:0/2442696598 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9433e00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434104ad0 0x7f943419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439c00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 0x7f94341a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 -- 172.21.15.123:0/2442696598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434104ad0 msgr2=0x7f943419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434104ad0 0x7f943419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 -- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 msgr2=0x7f94341a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 0x7f94341a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 -- 172.21.15.123:0/2442696598 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94341a1790 con 0x7f94341038e0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439c00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 0x7f94341a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9439200640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f942800e9c0 tx=0x7f942800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9431600640 1 -- 172.21.15.123:0/2442696598 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942800cd80 con 0x7f94341038e0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94341a1a80 con 0x7f94341038e0 2024-03-24T00:48:54.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9431600640 1 -- 172.21.15.123:0/2442696598 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94280035f0 con 0x7f94341038e0 2024-03-24T00:48:54.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.960+0000 7f9431600640 1 -- 172.21.15.123:0/2442696598 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428010690 con 0x7f94341038e0 2024-03-24T00:48:54.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.961+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94341a1f80 con 0x7f94341038e0 2024-03-24T00:48:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:54 smithi177 ceph-mon[36034]: pgmap v715: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.963+0000 7f943e83a740 1 Processor -- start 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 -- start start 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9431600640 1 -- 172.21.15.123:0/2442696598 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f94280037a0 con 0x7f94341038e0 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0246e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da024dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da01aa2c0 con 0x557da02469e0 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da0152c90 con 0x557da0245810 2024-03-24T00:48:54.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f943e83a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da01a6360 con 0x557da02473a0 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46594/0 (socket says 172.21.15.123:46594) 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 -- 172.21.15.123:0/3523890377 learned_addr learned my addr 172.21.15.123:0/3523890377 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439200640 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0246e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9433e00640 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da024dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 -- 172.21.15.123:0/3523890377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 msgr2=0x557da024dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da024dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 -- 172.21.15.123:0/3523890377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 msgr2=0x557da0246e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0246e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.964+0000 7f9439c00640 1 -- 172.21.15.123:0/3523890377 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557da0177080 con 0x557da0245810 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9439c00640 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f942c009ca0 tx=0x7f942c009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ed3875c87cab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9426000640 1 -- 172.21.15.123:0/3523890377 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c043020 con 0x557da0245810 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9426000640 1 -- 172.21.15.123:0/3523890377 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f942c003970 con 0x557da0245810 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9426000640 1 -- 172.21.15.123:0/3523890377 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c039040 con 0x557da0245810 2024-03-24T00:48:54.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9431600640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 0x7f941c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.965+0000 7f9431600640 1 -- 172.21.15.123:0/2442696598 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f94280ceec0 con 0x7f94341038e0 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f9439200640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 0x7f941c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- 172.21.15.123:0/3523890377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 msgr2=0x557da0245c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f942c009ca0 tx=0x7f942c009800 comp rx=0 tx=0).stop 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- 172.21.15.123:0/3523890377 shutdown_connections 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da024dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0246e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 --2- 172.21.15.123:0/3523890377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0245c30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- 172.21.15.123:0/3523890377 >> 172.21.15.123:0/3523890377 conn(0x557da0240c00 msgr2=0x557da0241000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- 172.21.15.123:0/3523890377 shutdown_connections 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- 172.21.15.123:0/3523890377 wait complete. 2024-03-24T00:48:54.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f9439200640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 0x7f941c0a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f94280cf160 tx=0x7f94280045f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.966+0000 7f943e83a740 1 -- start start 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 Processor -- start 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da0245810 0x557da024fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da0256fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da024e350 con 0x557da0245810 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da01660d0 con 0x557da02469e0 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da024e4d0 con 0x557da02473a0 2024-03-24T00:48:54.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f943e83a740 10 client.0 _ll_register_callbacks cb 0x557da016ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f9433e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f9439c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da0245810 0x557da024fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f9433e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46600/0 (socket says 172.21.15.123:46600) 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f9433e00640 1 -- 172.21.15.123:0/3539524497 learned_addr learned my addr 172.21.15.123:0/3539524497 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.967+0000 7f9439c00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da0256fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 -- 172.21.15.123:0/3539524497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 msgr2=0x557da0256fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da0256fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 -- 172.21.15.123:0/3539524497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da0245810 msgr2=0x557da024fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da0245810 0x557da024fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 -- 172.21.15.123:0/3539524497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d9febe3f0 con 0x557da02469e0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f9433e00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9420012700 tx=0x7f9420012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41f632760cfe425e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f941aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420004050 con 0x557da02469e0 2024-03-24T00:48:54.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f94200041f0 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f942c003970 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94200043f0 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f943e83a740 10 client.26738 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.968+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557d9ff887d0 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f942c003970 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9420003ca0 con 0x557da02469e0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 10 client.26738 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 10 client.26738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 20 client.26738 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da025b490 0x557da025d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 4 client.26738 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557da01b7800 con 0x557da025b490 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.969+0000 7f9439c00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da025b490 0x557da025d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.970+0000 7f9439c00640 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da025b490 0x557da025d910 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01843f851cf97bf server_cookie=1fda065c46478e90 in_seq=0 out_seq=0 2024-03-24T00:48:54.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.970+0000 7f941aa00640 10 client.26738 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.970+0000 7f941aa00640 1 -- 172.21.15.123:0/3539524497 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557da01b7800 con 0x557da025b490 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.970+0000 7f941aa00640 10 client.26738 handle_command_reply: tid=0 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.970+0000 7f943e83a740 1 client.26738 shutdown 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 msgr2=0x557da024ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9420012700 tx=0x7f9420012bc0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 shutdown_connections 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da025b490 0x557da025d910 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02473a0 0x557da0256fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02469e0 0x557da024ff70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f941aa00640 0 client.26738 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 --2- 172.21.15.123:0/3539524497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da0245810 0x557da024fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 >> 172.21.15.123:0/3539524497 conn(0x557da0240c00 msgr2=0x557da023feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 shutdown_connections 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 1 -- 172.21.15.123:0/3539524497 wait complete. 2024-03-24T00:48:54.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.971+0000 7f943e83a740 20 client.26738 trim_cache size 0 max 16384 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.986+0000 7f943e83a740 1 Processor -- start 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.986+0000 7f943e83a740 1 -- start start 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0256f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02473a0 0x557da02621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341a8cc0 con 0x557da02469e0 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341ab280 con 0x557da02473a0 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f943e83a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341a92c0 con 0x557da0245810 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:56474/0 (socket says 172.21.15.123:56474) 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 -- 172.21.15.123:0/1344713445 learned_addr learned my addr 172.21.15.123:0/1344713445 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02473a0 0x557da02621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9433e00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0256f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 -- 172.21.15.123:0/1344713445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02473a0 msgr2=0x557da02621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02473a0 0x557da02621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 -- 172.21.15.123:0/1344713445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 msgr2=0x557da0256f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0256f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.987+0000 7f9439c00640 1 -- 172.21.15.123:0/1344713445 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557da0246460 con 0x557da0245810 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.988+0000 7f9433e00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0256f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.988+0000 7f9439c00640 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f942c0324c0 tx=0x7f942c04c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c07e1dd02ddbc10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.988+0000 7f9425600640 1 -- 172.21.15.123:0/1344713445 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c039040 con 0x557da0245810 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.988+0000 7f9425600640 1 -- 172.21.15.123:0/1344713445 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f942c003e50 con 0x557da0245810 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.988+0000 7f9425600640 1 -- 172.21.15.123:0/1344713445 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c043040 con 0x557da0245810 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 -- 172.21.15.123:0/1344713445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 msgr2=0x557da023fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f942c0324c0 tx=0x7f942c04c040 comp rx=0 tx=0).stop 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 -- 172.21.15.123:0/1344713445 shutdown_connections 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da02473a0 0x557da02621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02469e0 0x557da0256f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 --2- 172.21.15.123:0/1344713445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da0245810 0x557da023fd30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 -- 172.21.15.123:0/1344713445 >> 172.21.15.123:0/1344713445 conn(0x557da0240c00 msgr2=0x557da023f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 -- 172.21.15.123:0/1344713445 shutdown_connections 2024-03-24T00:48:54.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.989+0000 7f943e83a740 1 -- 172.21.15.123:0/1344713445 wait complete. 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 -- start start 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 Processor -- start 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 0x557da0253950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02473a0 0x557da0253e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341a9900 con 0x557da02473a0 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341aac20 con 0x557da0245810 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341a9f60 con 0x557da02469e0 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.990+0000 7f943e83a740 10 client.0 _ll_register_callbacks cb 0x7f94200097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02473a0 0x557da0253e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:46622/0 (socket says 172.21.15.123:46622) 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 -- 172.21.15.123:0/768815152 learned_addr learned my addr 172.21.15.123:0/768815152 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:54.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9433e00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 0x557da0253950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 -- 172.21.15.123:0/768815152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 msgr2=0x557da0253950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 0x557da0253950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 -- 172.21.15.123:0/768815152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02473a0 msgr2=0x557da0253e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02473a0 0x557da0253e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 -- 172.21.15.123:0/768815152 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557da02bf5b0 con 0x557da0245810 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9433e00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 0x557da0253950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f9439c00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f942c02c640 tx=0x7f942c009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac12bad7531bbee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.991+0000 7f941aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c043070 con 0x557da0245810 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f942c032780 con 0x557da0245810 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f942c003e50 con 0x557da0245810 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f942c039040 con 0x557da0245810 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 10 client.26744 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:54.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557da02c0730 con 0x557da0245810 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f942c05a030 con 0x557da0245810 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f942c004560 con 0x557da0245810 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 10 client.26744 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 20 client.26744 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da02c0b10 0x557da02c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 4 client.26744 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.992+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557da01b1b50 con 0x557da02c0b10 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f9433e00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da02c0b10 0x557da02c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f941aa00640 10 client.26744 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f9433e00640 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da02c0b10 0x557da02c2f30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5938b49e4ed2b36f server_cookie=a21a52a25415a6 in_seq=0 out_seq=0 2024-03-24T00:48:54.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f941aa00640 1 -- 172.21.15.123:0/768815152 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557da01b1b50 con 0x557da02c0b10 2024-03-24T00:48:54.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f941aa00640 10 client.26744 handle_command_reply: tid=0 2024-03-24T00:48:54.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f943e83a740 1 client.26744 shutdown 2024-03-24T00:48:54.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 msgr2=0x557da0256e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.993+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f942c02c640 tx=0x7f942c009800 comp rx=0 tx=0).stop 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 shutdown_connections 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557da02473a0 0x557da0253e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x557da02c0b10 0x557da02c2f30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557da02469e0 0x557da0253950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 --2- 172.21.15.123:0/768815152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da0245810 0x557da0256e40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 >> 172.21.15.123:0/768815152 conn(0x557da0240c00 msgr2=0x557da023feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f941aa00640 0 client.26744 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 shutdown_connections 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 1 -- 172.21.15.123:0/768815152 wait complete. 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943e83a740 20 client.26744 trim_cache size 0 max 16384 2024-03-24T00:48:54.997 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 msgr2=0x7f941c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 0x7f941c0a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f94280cf160 tx=0x7f94280045f0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 msgr2=0x7f9434199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f942800e9c0 tx=0x7f942800ee80 comp rx=0 tx=0).stop 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 shutdown_connections 2024-03-24T00:48:54.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f941c0a2800 0x7f941c0a4cc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9434105490 0x7f94341a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9434104ad0 0x7f943419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 --2- 172.21.15.123:0/2442696598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94341038e0 0x7f9434199b70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.994+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 >> 172.21.15.123:0/2442696598 conn(0x7f94340fee50 msgr2=0x7f9434107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.995+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 shutdown_connections 2024-03-24T00:48:54.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:54.995+0000 7f943bc00640 1 -- 172.21.15.123:0/2442696598 wait complete. 2024-03-24T00:48:55.779 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2550112615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:55.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2550112615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:55.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2550112615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:48:56.013 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:48:56.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:56.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:48:56.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 -- 172.21.15.123:0/779032306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70104930 msgr2=0x7efd70106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 --2- 172.21.15.123:0/779032306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70104930 0x7efd70106d20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009900 tx=0x7efd5c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:48:56.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 -- 172.21.15.123:0/779032306 shutdown_connections 2024-03-24T00:48:56.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 --2- 172.21.15.123:0/779032306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70107260 0x7efd701096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 --2- 172.21.15.123:0/779032306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70104930 0x7efd70106d20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 --2- 172.21.15.123:0/779032306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70102000 0x7efd701043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 -- 172.21.15.123:0/779032306 >> 172.21.15.123:0/779032306 conn(0x7efd700fba10 msgr2=0x7efd700fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 -- 172.21.15.123:0/779032306 shutdown_connections 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.207+0000 7efd75e00640 1 -- 172.21.15.123:0/779032306 wait complete. 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.208+0000 7efd75e00640 1 Processor -- start 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.208+0000 7efd75e00640 1 -- start start 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 0x7efd70199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 0x7efd70199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 0x7efd701a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7010d260 con 0x7efd70102000 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7010d0e0 con 0x7efd70107260 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7010d3e0 con 0x7efd70104930 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 0x7efd70199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 0x7efd701a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 0x7efd70199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 0x7efd701a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45836/0 (socket says 172.21.15.123:45836) 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd75400640 1 -- 172.21.15.123:0/2810312535 learned_addr learned my addr 172.21.15.123:0/2810312535 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 -- 172.21.15.123:0/2810312535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 msgr2=0x7efd701a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 0x7efd701a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 -- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 msgr2=0x7efd70199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 0x7efd70199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd6fe00640 1 -- 172.21.15.123:0/2810312535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd701a16d0 con 0x7efd70104930 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.209+0000 7efd74a00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 0x7efd70199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 0x7efd70199ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009230 tx=0x7efd5c02c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd6d600640 1 -- 172.21.15.123:0/2810312535 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c043020 con 0x7efd70104930 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd6d600640 1 -- 172.21.15.123:0/2810312535 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd5c0054e0 con 0x7efd70104930 2024-03-24T00:48:56.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd701a1960 con 0x7efd70104930 2024-03-24T00:48:56.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd6d600640 1 -- 172.21.15.123:0/2810312535 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c003b60 con 0x7efd70104930 2024-03-24T00:48:56.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.210+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701a1e80 con 0x7efd70104930 2024-03-24T00:48:56.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 Processor -- start 2024-03-24T00:48:56.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 -- start start 2024-03-24T00:48:56.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd6d600640 1 -- 172.21.15.123:0/2810312535 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efd5c046020 con 0x7efd70104930 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4ee4eb0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4e8da60 con 0x555ef4f84e10 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.213+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4ee0f50 con 0x555ef4f7fca0 2024-03-24T00:48:56.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd75400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33506/0 (socket says 172.21.15.123:33506) 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 -- 172.21.15.123:0/1168582920 learned_addr learned my addr 172.21.15.123:0/1168582920 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 -- 172.21.15.123:0/1168582920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 msgr2=0x555ef4f872a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f872a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 -- 172.21.15.123:0/1168582920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 msgr2=0x555ef4f7f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 -- 172.21.15.123:0/1168582920 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ef4cc17e0 con 0x555ef4f7fca0 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd75400640 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.214+0000 7efd74a00640 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd64010550 tx=0x7efd64010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e187f597c405f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd61c00640 1 -- 172.21.15.123:0/1168582920 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64017070 con 0x555ef4f7fca0 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd61c00640 1 -- 172.21.15.123:0/1168582920 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd64011c40 con 0x555ef4f7fca0 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd61c00640 1 -- 172.21.15.123:0/1168582920 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6400ee50 con 0x555ef4f7fca0 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd6d600640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 0x7efd580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1168582920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 msgr2=0x555ef4f847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd64010550 tx=0x7efd64010a10 comp rx=0 tx=0).stop 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1168582920 shutdown_connections 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f872a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f847a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1168582920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1168582920 >> 172.21.15.123:0/1168582920 conn(0x555ef4f7a820 msgr2=0x555ef4f7ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd6d600640 1 -- 172.21.15.123:0/2810312535 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efd5c0f0b00 con 0x7efd70104930 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.215+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1168582920 shutdown_connections 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.216+0000 7efd75400640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 0x7efd580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.216+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1168582920 wait complete. 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.216+0000 7efd7a1b2740 1 -- start start 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.216+0000 7efd75400640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 0x7efd580a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efd68009640 tx=0x7efd68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.216+0000 7efd7a1b2740 1 Processor -- start 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f894c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4f87dc0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4ea0eb0 con 0x555ef4f84e10 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef4f87f40 con 0x555ef4f7fca0 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd7a1b2740 10 client.0 _ll_register_callbacks cb 0x555ef4e95150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45850/0 (socket says 172.21.15.123:45850) 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f894c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd75400640 1 -- 172.21.15.123:0/2720866285 learned_addr learned my addr 172.21.15.123:0/2720866285 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 -- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 msgr2=0x555ef4f90a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 -- 172.21.15.123:0/2720866285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 msgr2=0x555ef4f894c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f894c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd6fe00640 1 -- 172.21.15.123:0/2720866285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ef4d21f70 con 0x555ef4f7fca0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.217+0000 7efd75400640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd6fe00640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f89a00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0ee5b0 tx=0x7efd5c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30aa9ca15fb2d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd56000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0f9050 con 0x555ef4f7fca0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efd5c003d00 con 0x555ef4f7fca0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd64011c40 con 0x555ef4f7fca0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd5c0bcf90 con 0x555ef4f7fca0 2024-03-24T00:48:56.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd7a1b2740 10 client.26755 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555ef4bb9170 con 0x555ef4f7fca0 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.218+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd64011c40 con 0x555ef4f7fca0 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efd5c0b9ac0 con 0x555ef4f7fca0 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 10 client.26755 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 10 client.26755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 20 client.26755 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4f95310 0x555ef4f97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 4 client.26755 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ef4ef23f0 con 0x555ef4f95310 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd74a00640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4f95310 0x555ef4f97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd74a00640 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4f95310 0x555ef4f97760 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f4e3ed5f6c703ce server_cookie=24e790b4a9fcdad4 in_seq=0 out_seq=0 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.219+0000 7efd56000640 10 client.26755 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.220+0000 7efd56000640 1 -- 172.21.15.123:0/2720866285 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555ef4ef23f0 con 0x555ef4f95310 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.220+0000 7efd56000640 10 client.26755 handle_command_reply: tid=0 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.220+0000 7efd7a1b2740 1 client.26755 shutdown 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 msgr2=0x555ef4f89a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f89a00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0ee5b0 tx=0x7efd5c009260 comp rx=0 tx=0).stop 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 shutdown_connections 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f90a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4f95310 0x555ef4f97760 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f89a00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/2720866285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f894c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd56000640 0 client.26755 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 >> 172.21.15.123:0/2720866285 conn(0x555ef4f7a820 msgr2=0x555ef4f79a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 shutdown_connections 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 1 -- 172.21.15.123:0/2720866285 wait complete. 2024-03-24T00:48:56.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.221+0000 7efd7a1b2740 20 client.26755 trim_cache size 0 max 16384 2024-03-24T00:48:56.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 Processor -- start 2024-03-24T00:48:56.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 -- start start 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f90a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701a8a00 con 0x555ef4f7f2b0 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701aafc0 con 0x555ef4f84e10 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701a9000 con 0x555ef4f7fca0 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45864/0 (socket says 172.21.15.123:45864) 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd75400640 1 -- 172.21.15.123:0/1643305998 learned_addr learned my addr 172.21.15.123:0/1643305998 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.238+0000 7efd6fe00640 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f90a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 -- 172.21.15.123:0/1643305998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 msgr2=0x555ef4f90a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f90a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 -- 172.21.15.123:0/1643305998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 msgr2=0x555ef4f7e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 -- 172.21.15.123:0/1643305998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ef4f87ca0 con 0x555ef4f84e10 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd75400640 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7efd6801ea40 tx=0x7efd6801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2602afbcdcb4ec46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd61200640 1 -- 172.21.15.123:0/1643305998 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68005020 con 0x555ef4f84e10 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd61200640 1 -- 172.21.15.123:0/1643305998 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd680047a0 con 0x555ef4f84e10 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.239+0000 7efd61200640 1 -- 172.21.15.123:0/1643305998 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800f780 con 0x555ef4f84e10 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1643305998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 msgr2=0x555ef4f9c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7efd6801ea40 tx=0x7efd6801b000 comp rx=0 tx=0).stop 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1643305998 shutdown_connections 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f84e10 0x555ef4f9c000 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7fca0 0x555ef4f90a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/1643305998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f7f2b0 0x555ef4f7e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1643305998 >> 172.21.15.123:0/1643305998 conn(0x555ef4f7a820 msgr2=0x555ef4f79a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1643305998 shutdown_connections 2024-03-24T00:48:56.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- 172.21.15.123:0/1643305998 wait complete. 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 -- start start 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.240+0000 7efd7a1b2740 1 Processor -- start 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 0x555ef4f8efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 0x555ef4f8f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701a9640 con 0x555ef4f84e10 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701aa960 con 0x555ef4f7fca0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701a9ca0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd7a1b2740 10 client.0 _ll_register_callbacks cb 0x7efd5c0b90b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33540/0 (socket says 172.21.15.123:33540) 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 -- 172.21.15.123:0/3805689107 learned_addr learned my addr 172.21.15.123:0/3805689107 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 0x555ef4f8efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd75400640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 0x555ef4f8f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 -- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 msgr2=0x555ef4f8efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 0x555ef4f8efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 -- 172.21.15.123:0/3805689107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 msgr2=0x555ef4f8f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 0x555ef4f8f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd74a00640 1 -- 172.21.15.123:0/3805689107 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ef4ff0bd0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd6fe00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 0x555ef4f8efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.241+0000 7efd75400640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 0x555ef4f8f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd74a00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd640104a0 tx=0x7efd64009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28bf844bc80f8d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd56000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64017040 con 0x555ef4f7f2b0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd680047a0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efd64012e00 con 0x555ef4f7f2b0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd6402b210 con 0x555ef4f7f2b0 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd7a1b2740 10 client.26761 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:56.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555ef4ff1d70 con 0x555ef4f7f2b0 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.242+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd680047a0 con 0x555ef4f7f2b0 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efd6402b820 con 0x555ef4f7f2b0 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 10 client.26761 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 10 client.26761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 20 client.26761 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4ff2340 0x555ef4ff4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 4 client.26761 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555ef4eec740 con 0x555ef4ff2340 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd6fe00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4ff2340 0x555ef4ff4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd6fe00640 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4ff2340 0x555ef4ff4760 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0288df50c60194b server_cookie=3b24bd46e57acb7d in_seq=0 out_seq=0 2024-03-24T00:48:56.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.243+0000 7efd56000640 10 client.26761 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.244+0000 7efd56000640 1 -- 172.21.15.123:0/3805689107 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555ef4eec740 con 0x555ef4ff2340 2024-03-24T00:48:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.244+0000 7efd56000640 10 client.26761 handle_command_reply: tid=0 2024-03-24T00:48:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.244+0000 7efd7a1b2740 1 client.26761 shutdown 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 msgr2=0x555ef4f90990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd640104a0 tx=0x7efd64009c10 comp rx=0 tx=0).stop 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 shutdown_connections 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ef4f84e10 0x555ef4f8f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef4f7fca0 0x555ef4f8efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x555ef4ff2340 0x555ef4ff4760 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 --2- 172.21.15.123:0/3805689107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ef4f7f2b0 0x555ef4f90990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd56000640 0 client.26761 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 >> 172.21.15.123:0/3805689107 conn(0x555ef4f7a820 msgr2=0x555ef4f80740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 shutdown_connections 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 1 -- 172.21.15.123:0/3805689107 wait complete. 2024-03-24T00:48:56.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.245+0000 7efd7a1b2740 20 client.26761 trim_cache size 0 max 16384 2024-03-24T00:48:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:56.250 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:48:56.250 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "d3bdb2a8-6331-4fbb-9047-5e75cbf67d46", 2024-03-24T00:48:56.250 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:48:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 msgr2=0x7efd580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 0x7efd580a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efd68009640 tx=0x7efd68005f90 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 msgr2=0x7efd70199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 0x7efd70199ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009230 tx=0x7efd5c02c5d0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 shutdown_connections 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd70107260 0x7efd701a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd70104930 0x7efd70199ff0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7efd580a2800 0x7efd580a4cc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 --2- 172.21.15.123:0/2810312535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd70102000 0x7efd70199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 >> 172.21.15.123:0/2810312535 conn(0x7efd700fba10 msgr2=0x7efd70101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 shutdown_connections 2024-03-24T00:48:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.246+0000 7efd75e00640 1 -- 172.21.15.123:0/2810312535 wait complete. 2024-03-24T00:48:56.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:48:56.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 -- 172.21.15.123:0/83586088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0100880 msgr2=0x7f3ad0100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 --2- 172.21.15.123:0/83586088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0100880 0x7f3ad0100ce0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009860 tx=0x7f3abc02bb70 comp rx=0 tx=0).stop 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 -- 172.21.15.123:0/83586088 shutdown_connections 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 --2- 172.21.15.123:0/83586088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad0101220 0x7f3ad01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 --2- 172.21.15.123:0/83586088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0100880 0x7f3ad0100ce0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 --2- 172.21.15.123:0/83586088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad00fff20 0x7f3ad0100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 -- 172.21.15.123:0/83586088 >> 172.21.15.123:0/83586088 conn(0x7f3ad00fbb40 msgr2=0x7f3ad00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 -- 172.21.15.123:0/83586088 shutdown_connections 2024-03-24T00:48:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.461+0000 7f3ad5800640 1 -- 172.21.15.123:0/83586088 wait complete. 2024-03-24T00:48:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.462+0000 7f3ad5800640 1 Processor -- start 2024-03-24T00:48:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 -- start start 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad0100880 0x7f3ad019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0101220 0x7f3ad01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad010f4f0 con 0x7f3ad0100880 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad010f370 con 0x7f3ad00fff20 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ad5800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad010f670 con 0x7f3ad0101220 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45882/0 (socket says 172.21.15.123:45882) 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acea00640 1 -- 172.21.15.123:0/330314139 learned_addr learned my addr 172.21.15.123:0/330314139 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acf400640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0101220 0x7f3ad01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3ace000640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad0100880 0x7f3ad019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acea00640 1 -- 172.21.15.123:0/330314139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0101220 msgr2=0x7f3ad01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.463+0000 7f3acea00640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0101220 0x7f3ad01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3acea00640 1 -- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad0100880 msgr2=0x7f3ad019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3acea00640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad0100880 0x7f3ad019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3acea00640 1 -- 172.21.15.123:0/330314139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad01a17f0 con 0x7f3ad00fff20 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3acea00640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3ac7400640 1 -- 172.21.15.123:0/330314139 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000cd80 con 0x7f3ad00fff20 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad01a1ae0 con 0x7f3ad00fff20 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3ac7400640 1 -- 172.21.15.123:0/330314139 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac0010910 con 0x7f3ad00fff20 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3ac7400640 1 -- 172.21.15.123:0/330314139 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00188f0 con 0x7f3ad00fff20 2024-03-24T00:48:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.464+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad01a2050 con 0x7f3ad00fff20 2024-03-24T00:48:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.467+0000 7f3ac7400640 1 -- 172.21.15.123:0/330314139 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3ac0010ac0 con 0x7f3ad00fff20 2024-03-24T00:48:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.467+0000 7f3ad85fb740 1 Processor -- start 2024-03-24T00:48:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 -- start start 2024-03-24T00:48:56.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a6870 0x564c104ab370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104ab9e0 0x564c104ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c1040b250 con 0x564c104ab9e0 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c103b3a30 con 0x564c104a6870 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c104072f0 con 0x564c104a5e80 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3acea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a6870 0x564c104ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3acf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3acf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33568/0 (socket says 172.21.15.123:33568) 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3acf400640 1 -- 172.21.15.123:0/1028929412 learned_addr learned my addr 172.21.15.123:0/1028929412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.468+0000 7f3ace000640 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104ab9e0 0x564c104ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 -- 172.21.15.123:0/1028929412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a6870 msgr2=0x564c104ab370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a6870 0x564c104ab370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 -- 172.21.15.123:0/1028929412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104ab9e0 msgr2=0x564c104ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104ab9e0 0x564c104ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 -- 172.21.15.123:0/1028929412 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c1039e690 con 0x564c104a5e80 2024-03-24T00:48:56.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3ac7400640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 0x7f3ab40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acf400640 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800e450 tx=0x7f3ac80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bde5b1bcee5be79c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3ac7400640 1 -- 172.21.15.123:0/330314139 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ac00cee50 con 0x7f3ad00fff20 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3acea00640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 0x7f3ab40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3abbe00640 1 -- 172.21.15.123:0/1028929412 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800d910 con 0x564c104a5e80 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3abbe00640 1 -- 172.21.15.123:0/1028929412 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac801c070 con 0x564c104a5e80 2024-03-24T00:48:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.469+0000 7f3abbe00640 1 -- 172.21.15.123:0/1028929412 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8016380 con 0x564c104a5e80 2024-03-24T00:48:56.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 -- 172.21.15.123:0/1028929412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 msgr2=0x564c104a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800e450 tx=0x7f3ac80095c0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3acea00640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 0x7f3ab40a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3ac009b7e0 tx=0x7f3ac00d2890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 -- 172.21.15.123:0/1028929412 shutdown_connections 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104ab9e0 0x564c104ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a6870 0x564c104ab370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/1028929412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104a62a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 -- 172.21.15.123:0/1028929412 >> 172.21.15.123:0/1028929412 conn(0x564c1046e200 msgr2=0x564c104a17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 -- 172.21.15.123:0/1028929412 shutdown_connections 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.470+0000 7f3ad85fb740 1 -- 172.21.15.123:0/1028929412 wait complete. 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.471+0000 7f3ad85fb740 1 -- start start 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.471+0000 7f3ad85fb740 1 Processor -- start 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104afe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c104ae990 con 0x564c104a6870 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c103c6e80 con 0x564c104a5e80 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c104aeb10 con 0x564c104ab9e0 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ad85fb740 10 client.0 _ll_register_callbacks cb 0x564c103d9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3acf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3acf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45896/0 (socket says 172.21.15.123:45896) 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3acf400640 1 -- 172.21.15.123:0/2918420495 learned_addr learned my addr 172.21.15.123:0/2918420495 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3acf400640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 -- 172.21.15.123:0/2918420495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 msgr2=0x564c104b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 -- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 msgr2=0x564c104afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.472+0000 7f3ace000640 1 -- 172.21.15.123:0/2918420495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c101e97d0 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ace000640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b0360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3abc02c0b0 tx=0x7f3abc005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878f90d3269e2c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ab8c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc042070 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3abc004040 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ac801c070 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3abc04e050 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ad85fb740 10 client.16593 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.473+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564c1011f3f0 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac801c070 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3abc032640 con 0x564c104a6870 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 10 client.16593 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 10 client.16593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 20 client.16593 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c104bbb80 0x564c104bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 4 client.16593 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c10418790 con 0x564c104bbb80 2024-03-24T00:48:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.474+0000 7f3acf400640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c104bbb80 0x564c104bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.475+0000 7f3acf400640 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c104bbb80 0x564c104bdfb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64eba2ffc0878c1e server_cookie=c793e6225f87cc6 in_seq=0 out_seq=0 2024-03-24T00:48:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.475+0000 7f3ab8c00640 10 client.16593 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.475+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2918420495 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c10418790 con 0x564c104bbb80 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.475+0000 7f3ab8c00640 10 client.16593 handle_command_reply: tid=0 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.475+0000 7f3ad85fb740 1 client.16593 shutdown 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 msgr2=0x564c104b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b0360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3abc02c0b0 tx=0x7f3abc005b20 comp rx=0 tx=0).stop 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 shutdown_connections 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c104bbb80 0x564c104bdfb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b0360 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2918420495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 >> 172.21.15.123:0/2918420495 conn(0x564c1046e200 msgr2=0x564c104a0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ab8c00640 0 client.16593 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 shutdown_connections 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2918420495 wait complete. 2024-03-24T00:48:56.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.476+0000 7f3ad85fb740 20 client.16593 trim_cache size 0 max 16384 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.493+0000 7f3ad85fb740 1 Processor -- start 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.493+0000 7f3ad85fb740 1 -- start start 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01a8d10 con 0x564c104a6870 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01ab2d0 con 0x564c104a5e80 2024-03-24T00:48:56.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01a9310 con 0x564c104ab9e0 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3acf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3acf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45900/0 (socket says 172.21.15.123:45900) 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3acf400640 1 -- 172.21.15.123:0/864667272 learned_addr learned my addr 172.21.15.123:0/864667272 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3acf400640 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 -- 172.21.15.123:0/864667272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 msgr2=0x564c104c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 -- 172.21.15.123:0/864667272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 msgr2=0x564c104a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3ace000640 1 -- 172.21.15.123:0/864667272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c100df0f0 con 0x564c104a6870 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.494+0000 7f3acf400640 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.495+0000 7f3ace000640 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b7360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3abc032a60 tx=0x7f3abc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2357aab413eb099 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.495+0000 7f3abb400640 1 -- 172.21.15.123:0/864667272 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc04b020 con 0x564c104a6870 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.495+0000 7f3abb400640 1 -- 172.21.15.123:0/864667272 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3abc009450 con 0x564c104a6870 2024-03-24T00:48:56.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.495+0000 7f3abb400640 1 -- 172.21.15.123:0/864667272 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc042040 con 0x564c104a6870 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- 172.21.15.123:0/864667272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 msgr2=0x564c104b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b7360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3abc032a60 tx=0x7f3abc005bd0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- 172.21.15.123:0/864667272 shutdown_connections 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104ab9e0 0x564c104c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b7360 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/864667272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104a5e80 0x564c104a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- 172.21.15.123:0/864667272 >> 172.21.15.123:0/864667272 conn(0x564c1046e200 msgr2=0x564c104a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- 172.21.15.123:0/864667272 shutdown_connections 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- 172.21.15.123:0/864667272 wait complete. 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 -- start start 2024-03-24T00:48:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.496+0000 7f3ad85fb740 1 Processor -- start 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 0x564c104b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01a9950 con 0x564c104a6870 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01aac70 con 0x564c104ab9e0 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01a9fb0 con 0x564c104a5e80 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ad85fb740 10 client.0 _ll_register_callbacks cb 0x7f3ac0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3acf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3acf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104b5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:33608/0 (socket says 172.21.15.123:33608) 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3acf400640 1 -- 172.21.15.123:0/2303360741 learned_addr learned my addr 172.21.15.123:0/2303360741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3acf400640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 0x564c104b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 -- 172.21.15.123:0/2303360741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 msgr2=0x564c104b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 -- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 msgr2=0x564c104b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 0x564c104b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3ace000640 1 -- 172.21.15.123:0/2303360741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c104b6ed0 con 0x564c104a6870 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.497+0000 7f3acf400640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 0x564c104b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ace000640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b5990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x564c104a2790 tx=0x7f3abc0328e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ecf6da24d4a4f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ab8c00640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc056070 con 0x564c104a6870 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3abc004220 con 0x564c104a6870 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3abc009450 con 0x564c104a6870 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3abc04b030 con 0x564c104a6870 2024-03-24T00:48:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ad85fb740 10 client.16599 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.498+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564c10517d20 con 0x564c104a6870 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3abc009450 con 0x564c104a6870 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3abc059020 con 0x564c104a6870 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 10 client.16599 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 20 client.16599 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c10518600 0x564c1051aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 4 client.16599 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564c10412ae0 con 0x564c10518600 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.499+0000 7f3acf400640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c10518600 0x564c1051aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3acf400640 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c10518600 0x564c1051aa20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b002a9833d85ca99 server_cookie=1a866476f0424c42 in_seq=0 out_seq=0 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ab8c00640 10 client.16599 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ab8c00640 1 -- 172.21.15.123:0/2303360741 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564c10412ae0 con 0x564c10518600 2024-03-24T00:48:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ab8c00640 10 client.16599 handle_command_reply: tid=0 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 client.16599 shutdown 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 msgr2=0x564c104b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b5990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x564c104a2790 tx=0x7f3abc0328e0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 shutdown_connections 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564c104ab9e0 0x564c104b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564c104a6870 0x564c104b5990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564c10518600 0x564c1051aa20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 --2- 172.21.15.123:0/2303360741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c104a5e80 0x564c104b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 >> 172.21.15.123:0/2303360741 conn(0x564c1046e200 msgr2=0x564c104a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ab8c00640 0 client.16599 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 shutdown_connections 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 1 -- 172.21.15.123:0/2303360741 wait complete. 2024-03-24T00:48:56.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.500+0000 7f3ad85fb740 20 client.16599 trim_cache size 0 max 16384 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "d3bdb2a8-6331-4fbb-9047-5e75cbf67d46": { 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "d3bdb2a8-6331-4fbb-9047-5e75cbf67d46", 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:48:56.506 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:48:56.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 msgr2=0x7f3ab40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 0x7f3ab40a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3ac009b7e0 tx=0x7f3ac00d2890 comp rx=0 tx=0).stop 2024-03-24T00:48:56.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 msgr2=0x7f3ad0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 shutdown_connections 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad0101220 0x7f3ad01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3ab40a2800 0x7f3ab40a4cc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ad0100880 0x7f3ad019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 --2- 172.21.15.123:0/330314139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ad00fff20 0x7f3ad0199bd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 >> 172.21.15.123:0/330314139 conn(0x7f3ad00fbb40 msgr2=0x7f3ad00fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 shutdown_connections 2024-03-24T00:48:56.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:48:56.501+0000 7f3ad5800640 1 -- 172.21.15.123:0/330314139 wait complete. 2024-03-24T00:48:56.519 INFO:tasks.cephfs.filesystem:scrub status for tag:d3bdb2a8-6331-4fbb-9047-5e75cbf67d46 - {'path': '/', 'tag': 'd3bdb2a8-6331-4fbb-9047-5e75cbf67d46', 'options': 'recursive,force'} 2024-03-24T00:48:56.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:56 smithi204 ceph-mon[36345]: pgmap v716: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-03-24T00:48:56.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:56 smithi123 ceph-mon[31487]: pgmap v716: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-03-24T00:48:56.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:56 smithi177 ceph-mon[36034]: pgmap v716: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 KiB/s wr, 17 op/s 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 -- 172.21.15.123:0/2515345877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd580104ae0 msgr2=0x7fd580104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 --2- 172.21.15.123:0/2515345877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd580104ae0 0x7fd580104f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd568009900 tx=0x7fd56802bc30 comp rx=0 tx=0).stop 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 -- 172.21.15.123:0/2515345877 shutdown_connections 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 --2- 172.21.15.123:0/2515345877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5801054a0 0x7fd58010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 --2- 172.21.15.123:0/2515345877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd580104ae0 0x7fd580104f60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 --2- 172.21.15.123:0/2515345877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801038f0 0x7fd580103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.635+0000 7fd585000640 1 -- 172.21.15.123:0/2515345877 >> 172.21.15.123:0/2515345877 conn(0x7fd5800fee60 msgr2=0x7fd5801012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.636+0000 7fd585000640 1 -- 172.21.15.123:0/2515345877 shutdown_connections 2024-03-24T00:48:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.636+0000 7fd585000640 1 -- 172.21.15.123:0/2515345877 wait complete. 2024-03-24T00:48:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.636+0000 7fd585000640 1 Processor -- start 2024-03-24T00:48:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 -- start start 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd580104ae0 0x7fd58019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801054a0 0x7fd5801a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58010f5c0 con 0x7fd580104ae0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58010f440 con 0x7fd5801038f0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd585000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58010f740 con 0x7fd5801054a0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45924/0 (socket says 172.21.15.123:45924) 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 -- 172.21.15.123:0/2767894864 learned_addr learned my addr 172.21.15.123:0/2767894864 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57e800640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801054a0 0x7fd5801a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57d400640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd580104ae0 0x7fd58019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 -- 172.21.15.123:0/2767894864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801054a0 msgr2=0x7fd5801a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801054a0 0x7fd5801a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 -- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd580104ae0 msgr2=0x7fd58019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd580104ae0 0x7fd58019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.637+0000 7fd57de00640 1 -- 172.21.15.123:0/2767894864 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5801a17c0 con 0x7fd5801038f0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd57de00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd57000e9c0 tx=0x7fd57000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd576a00640 1 -- 172.21.15.123:0/2767894864 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57000ccc0 con 0x7fd5801038f0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd576a00640 1 -- 172.21.15.123:0/2767894864 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5700035f0 con 0x7fd5801038f0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5801a1ab0 con 0x7fd5801038f0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd576a00640 1 -- 172.21.15.123:0/2767894864 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57000ce60 con 0x7fd5801038f0 2024-03-24T00:48:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.638+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5801a1fd0 con 0x7fd5801038f0 2024-03-24T00:48:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.641+0000 7fd576a00640 1 -- 172.21.15.123:0/2767894864 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd5700037a0 con 0x7fd5801038f0 2024-03-24T00:48:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.641+0000 7fd587e2c740 1 Processor -- start 2024-03-24T00:48:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 -- start start 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x5562902296d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556290229d40 0x55629022c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562901e73b0 con 0x556290229d40 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556290189830 con 0x5562902241e0 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd587e2c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562901325d0 con 0x556290224bd0 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x5562902296d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45930/0 (socket says 172.21.15.123:45930) 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 -- 172.21.15.123:0/3644273230 learned_addr learned my addr 172.21.15.123:0/3644273230 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57d400640 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556290229d40 0x55629022c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 -- 172.21.15.123:0/3644273230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 msgr2=0x5562902296d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x5562902296d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 -- 172.21.15.123:0/3644273230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556290229d40 msgr2=0x55629022c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556290229d40 0x55629022c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.642+0000 7fd57e800640 1 -- 172.21.15.123:0/3644273230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55629014a2c0 con 0x5562902241e0 2024-03-24T00:48:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd57de00640 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x5562902296d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd57e800640 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd578011440 tx=0x7fd578011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7adc62a94a3307c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd56f400640 1 -- 172.21.15.123:0/3644273230 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578003c30 con 0x5562902241e0 2024-03-24T00:48:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd56f400640 1 -- 172.21.15.123:0/3644273230 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd578003dd0 con 0x5562902241e0 2024-03-24T00:48:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd56f400640 1 -- 172.21.15.123:0/3644273230 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57800fb50 con 0x5562902241e0 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd576a00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 0x7fd5640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd576a00640 1 -- 172.21.15.123:0/2767894864 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd5700cee40 con 0x7fd5801038f0 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.643+0000 7fd57de00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 0x7fd5640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 -- 172.21.15.123:0/3644273230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 msgr2=0x556290224600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd578011440 tx=0x7fd578011900 comp rx=0 tx=0).stop 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 -- 172.21.15.123:0/3644273230 shutdown_connections 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556290229d40 0x55629022c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x5562902296d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3644273230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5562902241e0 0x556290224600 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 -- 172.21.15.123:0/3644273230 >> 172.21.15.123:0/3644273230 conn(0x55629021f750 msgr2=0x55629021fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd57de00640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 0x7fd5640a4de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd570097bf0 tx=0x7fd5700d2880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 -- 172.21.15.123:0/3644273230 shutdown_connections 2024-03-24T00:48:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.644+0000 7fd587e2c740 1 -- 172.21.15.123:0/3644273230 wait complete. 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 -- start start 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 Processor -- start 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x55629022e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x5562902358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x556290145a20 con 0x5562902241e0 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562901858d0 con 0x556290224bd0 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55629022ccf0 con 0x556290229d40 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd587e2c740 10 client.0 _ll_register_callbacks cb 0x5562901821c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd57d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd57d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45944/0 (socket says 172.21.15.123:45944) 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd57d400640 1 -- 172.21.15.123:0/66999936 learned_addr learned my addr 172.21.15.123:0/66999936 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.645+0000 7fd57e800640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x55629022e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57e800640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x5562902358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 -- 172.21.15.123:0/66999936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 msgr2=0x5562902358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x5562902358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 -- 172.21.15.123:0/66999936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 msgr2=0x55629022e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x55629022e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 -- 172.21.15.123:0/66999936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55628fe9e210 con 0x556290224bd0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57e800640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x55629022e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd57d400640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd56802c8e0 tx=0x7fd568004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4480fec0cfa77202 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd563e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568005d50 con 0x556290224bd0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd568005230 con 0x556290224bd0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.646+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd578003dd0 con 0x556290224bd0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd568005430 con 0x556290224bd0 2024-03-24T00:48:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 10 client.26792 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55628ff6a620 con 0x556290224bd0 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd578003dd0 con 0x556290224bd0 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd5680037b0 con 0x556290224bd0 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 10 client.26792 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 10 client.26792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 20 client.26792 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55629023a160 0x55629023c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 4 client.26792 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556290196d70 con 0x55629023a160 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.647+0000 7fd57e800640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55629023a160 0x55629023c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.648+0000 7fd57e800640 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55629023a160 0x55629023c5c0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce4cac66de89d5b server_cookie=bce2ce94a948c503 in_seq=0 out_seq=0 2024-03-24T00:48:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.648+0000 7fd563e00640 10 client.26792 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd563e00640 1 -- 172.21.15.123:0/66999936 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556290196d70 con 0x55629023a160 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd563e00640 10 client.26792 handle_command_reply: tid=0 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 client.26792 shutdown 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 msgr2=0x55629022e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd56802c8e0 tx=0x7fd568004270 comp rx=0 tx=0).stop 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 shutdown_connections 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55629023a160 0x55629023c5c0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x5562902358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x55629022e910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd563e00640 0 client.26792 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 --2- 172.21.15.123:0/66999936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x55629022e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.649+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 >> 172.21.15.123:0/66999936 conn(0x55629021f750 msgr2=0x55629021e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.650+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 shutdown_connections 2024-03-24T00:48:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.650+0000 7fd587e2c740 1 -- 172.21.15.123:0/66999936 wait complete. 2024-03-24T00:48:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.650+0000 7fd587e2c740 20 client.26792 trim_cache size 0 max 16384 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.664+0000 7fd587e2c740 1 Processor -- start 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.664+0000 7fd587e2c740 1 -- start start 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x5562902229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x556290222ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801a8ce0 con 0x5562902241e0 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801ab2a0 con 0x556290224bd0 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd587e2c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801a92e0 con 0x556290229d40 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x5562902229b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x5562902229b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42026/0 (socket says 172.21.15.123:42026) 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57e800640 1 -- 172.21.15.123:0/3639329100 learned_addr learned my addr 172.21.15.123:0/3639329100 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:45958/0 (socket says 172.21.15.123:45958) 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57e800640 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x556290222ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 -- 172.21.15.123:0/3639329100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 msgr2=0x556290222ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x556290222ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 -- 172.21.15.123:0/3639329100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 msgr2=0x5562902229b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x5562902229b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57d400640 1 -- 172.21.15.123:0/3639329100 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556290225460 con 0x556290224bd0 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.665+0000 7fd57e800640 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x556290222ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd57d400640 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd568003ba0 tx=0x7fd568005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b478bce1f3a89d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd56ea00640 1 -- 172.21.15.123:0/3639329100 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568005d50 con 0x556290224bd0 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd56ea00640 1 -- 172.21.15.123:0/3639329100 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd568009780 con 0x556290224bd0 2024-03-24T00:48:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd56ea00640 1 -- 172.21.15.123:0/3639329100 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568034a70 con 0x556290224bd0 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd587e2c740 1 -- 172.21.15.123:0/3639329100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 msgr2=0x5562902358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.666+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd568003ba0 tx=0x7fd568005fd0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 -- 172.21.15.123:0/3639329100 shutdown_connections 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290229d40 0x556290222ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290224bd0 0x5562902358a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3639329100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x5562902229b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 -- 172.21.15.123:0/3639329100 >> 172.21.15.123:0/3639329100 conn(0x55629021f750 msgr2=0x556290225ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 -- 172.21.15.123:0/3639329100 shutdown_connections 2024-03-24T00:48:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 -- 172.21.15.123:0/3639329100 wait complete. 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 -- start start 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.667+0000 7fd587e2c740 1 Processor -- start 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x556290235840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290229d40 0x556290233910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801a9920 con 0x5562902241e0 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801aac40 con 0x556290229d40 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801a9f80 con 0x556290224bd0 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd587e2c740 10 client.0 _ll_register_callbacks cb 0x7fd570005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42030/0 (socket says 172.21.15.123:42030) 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57e800640 1 -- 172.21.15.123:0/3241942666 learned_addr learned my addr 172.21.15.123:0/3241942666 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x556290235840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57e800640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290229d40 0x556290233910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 -- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290229d40 msgr2=0x556290233910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290229d40 0x556290233910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 -- 172.21.15.123:0/3241942666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 msgr2=0x556290235300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57d400640 1 -- 172.21.15.123:0/3241942666 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556290234140 con 0x556290224bd0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.668+0000 7fd57e800640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd57d400640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x556290235840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x556290236f30 tx=0x7fd56802c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a3d95ac99662e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd563e00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568003e10 con 0x556290224bd0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd568009780 con 0x556290224bd0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd5680349c0 con 0x556290224bd0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd568004250 con 0x556290224bd0 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd587e2c740 10 client.26794 fetch_fsmap learned FSMap version 19 2024-03-24T00:48:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5562902964b0 con 0x556290224bd0 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.669+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd568009780 con 0x556290224bd0 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd568054bd0 con 0x556290224bd0 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 10 client.26794 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 10 client.26794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 20 client.26794 populate_metadata read hostname 'smithi123' 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556290296d60 0x556290299180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 4 client.26794 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5562901910c0 con 0x556290296d60 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd57e800640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556290296d60 0x556290299180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd563e00640 10 client.26794 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:48:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:48:57.670+0000 7fd57e800640 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556290296d60 0x556290299180 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56d4ed380ef199da server_cookie=5d2811f66ec7bae8 in_seq=0 out_seq=0 2024-03-24T00:48:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:57 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:48:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:57 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:57 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:48:57.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:48:57 smithi204 ceph-mon[36345]: pgmap v717: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 18 op/s 2024-03-24T00:48:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:57 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:48:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:57 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:57 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:48:57.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:48:57 smithi123 ceph-mon[31487]: pgmap v717: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 18 op/s 2024-03-24T00:48:57.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:57 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:48:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:57 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:48:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:57 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:48:57.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:48:57 smithi177 ceph-mon[36034]: pgmap v717: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 18 op/s 2024-03-24T00:49:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:00 smithi204 ceph-mon[36345]: pgmap v718: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-24T00:49:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:00 smithi123 ceph-mon[31487]: pgmap v718: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-24T00:49:00.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:00 smithi177 ceph-mon[36034]: pgmap v718: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-24T00:49:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:01 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:01.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:01 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:01.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:01 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:02 smithi204 ceph-mon[36345]: pgmap v719: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.4 KiB/s wr, 32 op/s 2024-03-24T00:49:02.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:02 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:49:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:02 smithi123 ceph-mon[31487]: pgmap v719: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.4 KiB/s wr, 32 op/s 2024-03-24T00:49:02.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:02 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:49:02.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:02 smithi177 ceph-mon[36034]: pgmap v719: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.4 KiB/s wr, 32 op/s 2024-03-24T00:49:02.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:02 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:49:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:04 smithi204 ceph-mon[36345]: pgmap v720: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 30 op/s 2024-03-24T00:49:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:04 smithi123 ceph-mon[31487]: pgmap v720: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 30 op/s 2024-03-24T00:49:04.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:04 smithi177 ceph-mon[36034]: pgmap v720: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 30 op/s 2024-03-24T00:49:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:06 smithi204 ceph-mon[36345]: pgmap v721: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.5 KiB/s wr, 33 op/s 2024-03-24T00:49:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:06 smithi123 ceph-mon[31487]: pgmap v721: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.5 KiB/s wr, 33 op/s 2024-03-24T00:49:06.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:06 smithi177 ceph-mon[36034]: pgmap v721: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.5 KiB/s wr, 33 op/s 2024-03-24T00:49:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.976+0000 7fd563e00640 1 -- 172.21.15.123:0/3241942666 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5562901910c0 con 0x556290296d60 2024-03-24T00:49:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.976+0000 7fd563e00640 10 client.26794 handle_command_reply: tid=0 2024-03-24T00:49:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.976+0000 7fd587e2c740 1 client.26794 shutdown 2024-03-24T00:49:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 msgr2=0x556290235840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x556290235840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x556290236f30 tx=0x7fd56802c2b0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 shutdown_connections 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556290229d40 0x556290233910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x556290296d60 0x556290299180 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556290224bd0 0x556290235840 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 --2- 172.21.15.123:0/3241942666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562902241e0 0x556290235300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 >> 172.21.15.123:0/3241942666 conn(0x55629021f750 msgr2=0x5562902229b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.977+0000 7fd563e00640 0 client.26794 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.978+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 shutdown_connections 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.978+0000 7fd587e2c740 1 -- 172.21.15.123:0/3241942666 wait complete. 2024-03-24T00:49:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.978+0000 7fd587e2c740 20 client.26794 trim_cache size 0 max 16384 2024-03-24T00:49:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:49:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:49:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:49:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 msgr2=0x7fd5640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 0x7fd5640a4de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd570097bf0 tx=0x7fd5700d2880 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 msgr2=0x7fd580199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd57000e9c0 tx=0x7fd57000edc0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 shutdown_connections 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5801054a0 0x7fd5801a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5640a2920 0x7fd5640a4de0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd580104ae0 0x7fd58019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 --2- 172.21.15.123:0/2767894864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5801038f0 0x7fd580199ba0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 >> 172.21.15.123:0/2767894864 conn(0x7fd5800fee60 msgr2=0x7fd580107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.979+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 shutdown_connections 2024-03-24T00:49:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:06.980+0000 7fd585000640 1 -- 172.21.15.123:0/2767894864 wait complete. 2024-03-24T00:49:08.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:08 smithi204 ceph-mon[36345]: pgmap v722: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 KiB/s wr, 36 op/s 2024-03-24T00:49:08.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:08 smithi123 ceph-mon[31487]: pgmap v722: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 KiB/s wr, 36 op/s 2024-03-24T00:49:08.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:08 smithi177 ceph-mon[36034]: pgmap v722: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 KiB/s wr, 36 op/s 2024-03-24T00:49:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:10 smithi177 ceph-mon[36034]: pgmap v723: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 KiB/s wr, 33 op/s 2024-03-24T00:49:10.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:10 smithi204 ceph-mon[36345]: pgmap v723: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 KiB/s wr, 33 op/s 2024-03-24T00:49:10.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:10 smithi123 ceph-mon[31487]: pgmap v723: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 KiB/s wr, 33 op/s 2024-03-24T00:49:11.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:11 smithi204 ceph-mon[36345]: pgmap v724: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 27 op/s 2024-03-24T00:49:11.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:11 smithi123 ceph-mon[31487]: pgmap v724: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 27 op/s 2024-03-24T00:49:11.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:11 smithi177 ceph-mon[36034]: pgmap v724: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 27 op/s 2024-03-24T00:49:14.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:14 smithi204 ceph-mon[36345]: pgmap v725: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 45 KiB/s wr, 17 op/s 2024-03-24T00:49:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:14 smithi123 ceph-mon[31487]: pgmap v725: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 45 KiB/s wr, 17 op/s 2024-03-24T00:49:14.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:14 smithi177 ceph-mon[36034]: pgmap v725: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 45 KiB/s wr, 17 op/s 2024-03-24T00:49:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:16 smithi204 ceph-mon[36345]: pgmap v726: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-03-24T00:49:16.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:16 smithi123 ceph-mon[31487]: pgmap v726: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-03-24T00:49:16.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:16 smithi177 ceph-mon[36034]: pgmap v726: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-03-24T00:49:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:18 smithi204 ceph-mon[36345]: pgmap v727: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-03-24T00:49:18.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:18 smithi123 ceph-mon[31487]: pgmap v727: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-03-24T00:49:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:18 smithi177 ceph-mon[36034]: pgmap v727: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 -- 172.21.15.123:0/1944437985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 msgr2=0x7f1a6c073af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 --2- 172.21.15.123:0/1944437985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c073af0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009900 tx=0x7f1a5c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 -- 172.21.15.123:0/1944437985 shutdown_connections 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 --2- 172.21.15.123:0/1944437985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c109580 0x7f1a6c10b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 --2- 172.21.15.123:0/1944437985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c0740c0 0x7f1a6c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 --2- 172.21.15.123:0/1944437985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c073af0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 -- 172.21.15.123:0/1944437985 >> 172.21.15.123:0/1944437985 conn(0x7f1a6c0fcc60 msgr2=0x7f1a6c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 -- 172.21.15.123:0/1944437985 shutdown_connections 2024-03-24T00:49:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.123+0000 7f1a72c00640 1 -- 172.21.15.123:0/1944437985 wait complete. 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.124+0000 7f1a72c00640 1 Processor -- start 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.124+0000 7f1a72c00640 1 -- start start 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c0740c0 0x7f1a6c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10f580 con 0x7f1a6c0756a0 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10f400 con 0x7f1a6c0740c0 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a72c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10f700 con 0x7f1a6c109580 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c0740c0 0x7f1a6c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50366/0 (socket says 172.21.15.123:50366) 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a63e00640 1 -- 172.21.15.123:0/3983472497 learned_addr learned my addr 172.21.15.123:0/3983472497 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50096/0 (socket says 172.21.15.123:50096) 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 -- 172.21.15.123:0/3983472497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c0740c0 msgr2=0x7f1a6c199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c0740c0 0x7f1a6c199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 -- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 msgr2=0x7f1a6c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.125+0000 7f1a70c00640 1 -- 172.21.15.123:0/3983472497 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c1a1730 con 0x7f1a6c109580 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a70c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400ec50 tx=0x7f1a6400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a69600640 1 -- 172.21.15.123:0/3983472497 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a64003b70 con 0x7f1a6c109580 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a69600640 1 -- 172.21.15.123:0/3983472497 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a64003d10 con 0x7f1a6c109580 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c1a1a20 con 0x7f1a6c109580 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a69600640 1 -- 172.21.15.123:0/3983472497 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a64010890 con 0x7f1a6c109580 2024-03-24T00:49:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.126+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c1a1f20 con 0x7f1a6c109580 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.129+0000 7f1a75974740 1 Processor -- start 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.129+0000 7f1a69600640 1 -- 172.21.15.123:0/3983472497 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1a64003fc0 con 0x7f1a6c109580 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 -- start start 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f60b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f60e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f5c93b0 con 0x56268f606bd0 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f56b830 con 0x56268f6061e0 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a75974740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f5145d0 con 0x56268f60bd40 2024-03-24T00:49:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:35634/0 (socket says 172.21.15.123:35634) 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a63e00640 1 -- 172.21.15.123:0/2281065481 learned_addr learned my addr 172.21.15.123:0/2281065481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f60e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.130+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f60b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 -- 172.21.15.123:0/2281065481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 msgr2=0x56268f60e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f60e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 -- 172.21.15.123:0/2281065481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 msgr2=0x56268f60b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f60b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 -- 172.21.15.123:0/2281065481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268f52c2c0 con 0x56268f6061e0 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f60b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a540118f0 tx=0x7f1a540098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36458bf2c6b88836 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a61200640 1 -- 172.21.15.123:0/2281065481 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400f7c0 con 0x56268f6061e0 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a69600640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 0x7f1a500a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a61200640 1 -- 172.21.15.123:0/2281065481 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a5400fe40 con 0x56268f6061e0 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a61200640 1 -- 172.21.15.123:0/2281065481 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400eda0 con 0x56268f6061e0 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.131+0000 7f1a69600640 1 -- 172.21.15.123:0/3983472497 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1a640ceff0 con 0x7f1a6c109580 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a6be00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 0x7f1a500a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 -- 172.21.15.123:0/2281065481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 msgr2=0x56268f606600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a540118f0 tx=0x7f1a540098d0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 -- 172.21.15.123:0/2281065481 shutdown_connections 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f60e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f60b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 --2- 172.21.15.123:0/2281065481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f6061e0 0x56268f606600 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 -- 172.21.15.123:0/2281065481 >> 172.21.15.123:0/2281065481 conn(0x56268f601750 msgr2=0x56268f601b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 -- 172.21.15.123:0/2281065481 shutdown_connections 2024-03-24T00:49:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a75974740 1 -- 172.21.15.123:0/2281065481 wait complete. 2024-03-24T00:49:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.132+0000 7f1a6be00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 0x7f1a500a4d60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c02c240 tx=0x7f1a5c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 -- start start 2024-03-24T00:49:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 Processor -- start 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 0x56268f6103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f6108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f617920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f527a20 con 0x56268f606bd0 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f5678d0 con 0x56268f60bd40 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268f60ecf0 con 0x56268f6061e0 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.133+0000 7f1a75974740 10 client.0 _ll_register_callbacks cb 0x56268f5641c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 0x56268f6103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f6108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f617920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 0x56268f6103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50112/0 (socket says 172.21.15.123:50112) 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a6be00640 1 -- 172.21.15.123:0/2497352166 learned_addr learned my addr 172.21.15.123:0/2497352166 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 -- 172.21.15.123:0/2497352166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 msgr2=0x56268f6103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 0x56268f6103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 -- 172.21.15.123:0/2497352166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 msgr2=0x56268f6108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f6108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 -- 172.21.15.123:0/2497352166 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56268f280210 con 0x56268f60bd40 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a70c00640 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f617920 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1a6409b910 tx=0x7f1a6409b580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36897a881da6923 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a59600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a640b4060 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1a640a3e90 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.134+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a5400fe40 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a6409b980 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 10 client.26810 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56268f34c620 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a5400fe40 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a64004b40 con 0x56268f60bd40 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 10 client.26810 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 10 client.26810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 20 client.26810 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f61c150 0x56268f61e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 4 client.26810 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56268f578d70 con 0x56268f61c150 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.135+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f61c150 0x56268f61e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.136+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f61c150 0x56268f61e540 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c91c729bb102a3a server_cookie=b2116c2f63685813 in_seq=0 out_seq=0 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.136+0000 7f1a59600640 10 client.26810 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a59600640 1 -- 172.21.15.123:0/2497352166 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56268f578d70 con 0x56268f61c150 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a59600640 10 client.26810 handle_command_reply: tid=0 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 client.26810 shutdown 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 msgr2=0x56268f617920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f617920 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1a6409b910 tx=0x7f1a6409b580 comp rx=0 tx=0).stop 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 shutdown_connections 2024-03-24T00:49:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f617920 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f606bd0 0x56268f6108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f61c150 0x56268f61e540 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 --2- 172.21.15.123:0/2497352166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f6061e0 0x56268f6103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a59600640 0 client.26810 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 >> 172.21.15.123:0/2497352166 conn(0x56268f601750 msgr2=0x56268f600950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 shutdown_connections 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 1 -- 172.21.15.123:0/2497352166 wait complete. 2024-03-24T00:49:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.137+0000 7f1a75974740 20 client.26810 trim_cache size 0 max 16384 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 Processor -- start 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 -- start start 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f6049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 0x56268f6178e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f604ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a8c00 con 0x56268f6061e0 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1ab1c0 con 0x56268f606bd0 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.154+0000 7f1a75974740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a9200 con 0x56268f60bd40 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f6049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a70c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f604ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 0x56268f6178e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f6049b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50408/0 (socket says 172.21.15.123:50408) 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a70c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f604ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50124/0 (socket says 172.21.15.123:50124) 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a6be00640 1 -- 172.21.15.123:0/2423464533 learned_addr learned my addr 172.21.15.123:0/2423464533 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 -- 172.21.15.123:0/2423464533 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 msgr2=0x56268f604ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f604ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 -- 172.21.15.123:0/2423464533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 msgr2=0x56268f6049b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f6049b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 -- 172.21.15.123:0/2423464533 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268f60ee30 con 0x56268f606bd0 2024-03-24T00:49:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.155+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 0x56268f6178e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1a540098a0 tx=0x7f1a54009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42c2d13efd9ade8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a5be00640 1 -- 172.21.15.123:0/2423464533 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54009de0 con 0x56268f606bd0 2024-03-24T00:49:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a5be00640 1 -- 172.21.15.123:0/2423464533 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a54005510 con 0x56268f606bd0 2024-03-24T00:49:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a5be00640 1 -- 172.21.15.123:0/2423464533 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540048e0 con 0x56268f606bd0 2024-03-24T00:49:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 -- 172.21.15.123:0/2423464533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 msgr2=0x56268f6178e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 0x56268f6178e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1a540098a0 tx=0x7f1a54009530 comp rx=0 tx=0).stop 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 -- 172.21.15.123:0/2423464533 shutdown_connections 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f60bd40 0x56268f604ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f606bd0 0x56268f6178e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 --2- 172.21.15.123:0/2423464533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f6049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 -- 172.21.15.123:0/2423464533 >> 172.21.15.123:0/2423464533 conn(0x56268f601750 msgr2=0x56268f601b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 -- 172.21.15.123:0/2423464533 shutdown_connections 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.156+0000 7f1a75974740 1 -- 172.21.15.123:0/2423464533 wait complete. 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 -- start start 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 Processor -- start 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f617690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f6161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a9840 con 0x56268f6061e0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1aab60 con 0x56268f60bd40 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a9ea0 con 0x56268f606bd0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.157+0000 7f1a75974740 10 client.0 _ll_register_callbacks cb 0x7f1a64003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f617690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:50134/0 (socket says 172.21.15.123:50134) 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 -- 172.21.15.123:0/2188038380 learned_addr learned my addr 172.21.15.123:0/2188038380 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a70c00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f6161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 -- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 msgr2=0x56268f6161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f6161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 -- 172.21.15.123:0/2188038380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 msgr2=0x56268f617690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f617690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 -- 172.21.15.123:0/2188038380 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56268f610c50 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a63e00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009cb0 tx=0x7f1a54009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19f58c1da7ae36f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a59600640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.158+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54037020 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a54005510 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1a54005230 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a5400fea0 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 10 client.26812 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56268f678630 con 0x56268f606bd0 2024-03-24T00:49:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a54005510 con 0x56268f606bd0 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a5401d020 con 0x56268f606bd0 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 10 client.26812 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 10 client.26812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 20 client.26812 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f678d40 0x56268f67b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 4 client.26812 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.159+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56268f5730c0 con 0x56268f678d40 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.160+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f678d40 0x56268f67b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.160+0000 7f1a6be00640 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f678d40 0x56268f67b160 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75979996916c29de server_cookie=54d4682a36b53966 in_seq=0 out_seq=0 2024-03-24T00:49:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:20.160+0000 7f1a59600640 10 client.26812 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:20.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:20 smithi204 ceph-mon[36345]: pgmap v728: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-03-24T00:49:20.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:20 smithi123 ceph-mon[31487]: pgmap v728: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-03-24T00:49:20.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:20 smithi177 ceph-mon[36034]: pgmap v728: 97 pgs: 97 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-03-24T00:49:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: pgmap v729: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-03-24T00:49:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:22.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:22 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: pgmap v729: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:22 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: pgmap v729: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-03-24T00:49:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:22.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:22 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:23.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:23.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:23.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:23.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:23.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:23.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:49:24.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:24 smithi204 ceph-mon[36345]: pgmap v730: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:24.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:24 smithi123 ceph-mon[31487]: pgmap v730: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:24.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:24 smithi177 ceph-mon[36034]: pgmap v730: 97 pgs: 97 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:26.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:49:26.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:49:26.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 -- 172.21.15.123:0/219545733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 msgr2=0x7f23c01109a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 --2- 172.21.15.123:0/219545733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c01109a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009ca0 tx=0x7f23b802c100 comp rx=0 tx=0).stop 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 -- 172.21.15.123:0/219545733 shutdown_connections 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 --2- 172.21.15.123:0/219545733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c01109a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 --2- 172.21.15.123:0/219545733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 0x7f23c0109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 --2- 172.21.15.123:0/219545733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 0x7f23c0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 -- 172.21.15.123:0/219545733 >> 172.21.15.123:0/219545733 conn(0x7f23c006e8a0 msgr2=0x7f23c0070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.741+0000 7f23c7600640 1 -- 172.21.15.123:0/219545733 shutdown_connections 2024-03-24T00:49:26.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.742+0000 7f23c7600640 1 -- 172.21.15.123:0/219545733 wait complete. 2024-03-24T00:49:26.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.742+0000 7f23c7600640 1 Processor -- start 2024-03-24T00:49:26.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 -- start start 2024-03-24T00:49:26.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 0x7f23c010c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 0x7f23c010ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0113930 con 0x7f23c0109750 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01137b0 con 0x7f23c010a0f0 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c7600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0113ab0 con 0x7f23c0108a90 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 0x7f23c010ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:32942/0 (socket says 172.21.15.123:32942) 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c5600640 1 -- 172.21.15.123:0/3836217133 learned_addr learned my addr 172.21.15.123:0/3836217133 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.743+0000 7f23c4c00640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 0x7f23c010c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 -- 172.21.15.123:0/3836217133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 msgr2=0x7f23c010c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 0x7f23c010c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 -- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 msgr2=0x7f23c010ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:26.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 0x7f23c010ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 -- 172.21.15.123:0/3836217133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c010b9d0 con 0x7f23c010a0f0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c5600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f23b802c640 tx=0x7f23b8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23bd600640 1 -- 172.21.15.123:0/3836217133 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8039040 con 0x7f23c010a0f0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23bd600640 1 -- 172.21.15.123:0/3836217133 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b8044070 con 0x7f23c010a0f0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23bd600640 1 -- 172.21.15.123:0/3836217133 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80043c0 con 0x7f23c010a0f0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c01aa200 con 0x7f23c010a0f0 2024-03-24T00:49:26.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.744+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c01aa700 con 0x7f23c010a0f0 2024-03-24T00:49:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.747+0000 7f23bd600640 1 -- 172.21.15.123:0/3836217133 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f23b8004560 con 0x7f23c010a0f0 2024-03-24T00:49:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.747+0000 7f23ca393740 1 Processor -- start 2024-03-24T00:49:26.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 -- start start 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc4093370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc4095e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc3ff3250 con 0x55efc40939e0 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc3f9ba30 con 0x55efc408de80 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23ca393740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc3fef2f0 con 0x55efc408e870 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc4093370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:32948/0 (socket says 172.21.15.123:32948) 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23bfe00640 1 -- 172.21.15.123:0/4002088652 learned_addr learned my addr 172.21.15.123:0/4002088652 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23bfe00640 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc4095e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.748+0000 7f23c4c00640 1 -- 172.21.15.123:0/4002088652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 msgr2=0x55efc408e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:26.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23c4c00640 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23c4c00640 1 -- 172.21.15.123:0/4002088652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 msgr2=0x55efc4095e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23c4c00640 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc4095e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23c4c00640 1 -- 172.21.15.123:0/4002088652 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efc3f86690 con 0x55efc408e870 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23bfe00640 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23c4c00640 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc4093370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009c00 tx=0x7f23b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629aa64223c98277 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23bd600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 0x7f23a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23adc00640 1 -- 172.21.15.123:0/4002088652 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4004230 con 0x55efc408e870 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23adc00640 1 -- 172.21.15.123:0/4002088652 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b40043d0 con 0x55efc408e870 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23adc00640 1 -- 172.21.15.123:0/4002088652 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4004d20 con 0x55efc408e870 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23bd600640 1 -- 172.21.15.123:0/3836217133 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f23b80f0fe0 con 0x7f23c010a0f0 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.749+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 0x7f23a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 -- 172.21.15.123:0/4002088652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 msgr2=0x55efc4093370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc4093370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009c00 tx=0x7f23b4011810 comp rx=0 tx=0).stop 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 0x7f23a80a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f23b000afd0 tx=0x7f23b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:26.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 -- 172.21.15.123:0/4002088652 shutdown_connections 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc4095e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc4093370 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 --2- 172.21.15.123:0/4002088652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc408e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 -- 172.21.15.123:0/4002088652 >> 172.21.15.123:0/4002088652 conn(0x55efc4056200 msgr2=0x55efc40897f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 -- 172.21.15.123:0/4002088652 shutdown_connections 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.750+0000 7f23ca393740 1 -- 172.21.15.123:0/4002088652 wait complete. 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 -- start start 2024-03-24T00:49:26.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 Processor -- start 2024-03-24T00:49:26.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc4098390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc4096990 con 0x55efc40939e0 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc3faee80 con 0x55efc408e870 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc4096b10 con 0x55efc408de80 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23ca393740 10 client.0 _ll_register_callbacks cb 0x55efc3fc1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.751+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57594/0 (socket says 172.21.15.123:57594) 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 -- 172.21.15.123:0/3179041352 learned_addr learned my addr 172.21.15.123:0/3179041352 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc4098390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c5600640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 -- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 msgr2=0x55efc4098390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc4098390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 -- 172.21.15.123:0/3179041352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 msgr2=0x55efc409f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 -- 172.21.15.123:0/3179041352 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efc3dd17d0 con 0x55efc408de80 2024-03-24T00:49:26.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c5600640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23c4c00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23b40096e0 tx=0x7f23b4019a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739a56a8e4f746ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23a2000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.752+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4028020 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f23b4024620 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23b40043d0 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23b4019e50 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23ca393740 10 client.26824 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55efc3d073f0 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.753+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b40043d0 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f23b4018070 con 0x55efc408de80 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 10 client.26824 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 20 client.26824 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40a3960 0x55efc40a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 4 client.26824 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efc4000790 con 0x55efc40a3960 2024-03-24T00:49:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.754+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40a3960 0x55efc40a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.755+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40a3960 0x55efc40a5d90 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16e1dccdb14796ac server_cookie=4c6c936a26d32fa1 in_seq=0 out_seq=0 2024-03-24T00:49:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:26.755+0000 7f23a2000640 10 client.26824 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:26 smithi204 ceph-mon[36345]: pgmap v731: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 16 op/s 2024-03-24T00:49:26.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:26 smithi123 ceph-mon[31487]: pgmap v731: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 16 op/s 2024-03-24T00:49:26.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:26 smithi177 ceph-mon[36034]: pgmap v731: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 16 op/s 2024-03-24T00:49:28.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:28 smithi204 ceph-mon[36345]: pgmap v732: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:28.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:28 smithi123 ceph-mon[31487]: pgmap v732: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:28.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:28 smithi177 ceph-mon[36034]: pgmap v732: 97 pgs: 97 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-03-24T00:49:29.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:29 smithi204 ceph-mon[36345]: pgmap v733: 97 pgs: 97 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-03-24T00:49:29.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:29 smithi123 ceph-mon[31487]: pgmap v733: 97 pgs: 97 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-03-24T00:49:29.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:29 smithi177 ceph-mon[36034]: pgmap v733: 97 pgs: 97 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-03-24T00:49:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.836+0000 7f1a59600640 1 -- 172.21.15.123:0/2188038380 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56268f5730c0 con 0x56268f678d40 2024-03-24T00:49:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.836+0000 7f1a59600640 10 client.26812 handle_command_reply: tid=0 2024-03-24T00:49:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 client.26812 shutdown 2024-03-24T00:49:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f23a2000640 1 -- 172.21.15.123:0/3179041352 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55efc4000790 con 0x55efc40a3960 2024-03-24T00:49:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f23a2000640 10 client.26824 handle_command_reply: tid=0 2024-03-24T00:49:31.838 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 msgr2=0x56268f615ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f23ca393740 1 client.26824 shutdown 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009cb0 tx=0x7f1a54009370 comp rx=0 tx=0).stop 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 shutdown_connections 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56268f60bd40 0x56268f6161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x56268f678d40 0x56268f67b160 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56268f606bd0 0x56268f615ca0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 --2- 172.21.15.123:0/2188038380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56268f6061e0 0x56268f617690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 >> 172.21.15.123:0/2188038380 conn(0x56268f601750 msgr2=0x56268f6049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f1a59600640 0 client.26812 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 msgr2=0x55efc4097e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.837+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23b40096e0 tx=0x7f23b4019a50 comp rx=0 tx=0).stop 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 shutdown_connections 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc4098390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40a3960 0x55efc40a5d90 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 --2- 172.21.15.123:0/3179041352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408de80 0x55efc4097e50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 >> 172.21.15.123:0/3179041352 conn(0x55efc4056200 msgr2=0x55efc4088a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23a2000640 0 client.26824 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:31.841 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 shutdown_connections 2024-03-24T00:49:31.841 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f1a75974740 1 -- 172.21.15.123:0/2188038380 wait complete. 2024-03-24T00:49:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 shutdown_connections 2024-03-24T00:49:31.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 1 -- 172.21.15.123:0/3179041352 wait complete. 2024-03-24T00:49:31.842 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f1a75974740 20 client.26812 trim_cache size 0 max 16384 2024-03-24T00:49:31.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.838+0000 7f23ca393740 20 client.26824 trim_cache size 0 max 16384 2024-03-24T00:49:31.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:{ 2024-03-24T00:49:31.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "message": "", 2024-03-24T00:49:31.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout: "return_code": 0 2024-03-24T00:49:31.843 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stdout:} 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.839+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 msgr2=0x7f1a500a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.839+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 0x7f1a500a4d60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c02c240 tx=0x7f1a5c02df90 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.839+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 msgr2=0x7f1a6c1a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.839+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400ec50 tx=0x7f1a6400c6a0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 shutdown_connections 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a6c109580 0x7f1a6c1a10c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f1a500a28a0 0x7f1a500a4d60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c0756a0 0x7f1a6c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 --2- 172.21.15.123:0/3983472497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6c0740c0 0x7f1a6c199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.844 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 >> 172.21.15.123:0/3983472497 conn(0x7f1a6c0fcc60 msgr2=0x7f1a6c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 shutdown_connections 2024-03-24T00:49:31.845 INFO:teuthology.task.background_exec.ubuntu@smithi123.front.sepia.ceph.com.smithi123.stderr:2024-03-24T00:49:31.840+0000 7f1a72c00640 1 -- 172.21.15.123:0/3983472497 wait complete. 2024-03-24T00:49:31.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.860+0000 7f23ca393740 1 Processor -- start 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.860+0000 7f23ca393740 1 -- start start 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 0x55efc408ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc409f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b1450 con 0x55efc408de80 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b3a10 con 0x55efc408e870 2024-03-24T00:49:31.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23ca393740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b1a50 con 0x55efc40939e0 2024-03-24T00:49:31.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23c5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 0x55efc408ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23c5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57596/0 (socket says 172.21.15.123:57596) 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23c5600640 1 -- 172.21.15.123:0/3779693171 learned_addr learned my addr 172.21.15.123:0/3779693171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.861+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc409f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 -- 172.21.15.123:0/3779693171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 msgr2=0x55efc409f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc409f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 -- 172.21.15.123:0/3779693171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 msgr2=0x55efc408ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 0x55efc408ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 -- 172.21.15.123:0/3779693171 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efc3cc70f0 con 0x55efc40939e0 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23bfe00640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc409f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c4c00640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 0x55efc408ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23c5600640 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f23b8005e30 tx=0x7f23b80ee960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47ea8083c98dc0e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:31.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23ad200640 1 -- 172.21.15.123:0/3779693171 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80098b0 con 0x55efc40939e0 2024-03-24T00:49:31.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23ad200640 1 -- 172.21.15.123:0/3779693171 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b80bd930 con 0x55efc40939e0 2024-03-24T00:49:31.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.862+0000 7f23ad200640 1 -- 172.21.15.123:0/3779693171 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80bdb90 con 0x55efc40939e0 2024-03-24T00:49:31.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 -- 172.21.15.123:0/3779693171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 msgr2=0x55efc408cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f23b8005e30 tx=0x7f23b80ee960 comp rx=0 tx=0).stop 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 -- 172.21.15.123:0/3779693171 shutdown_connections 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc40939e0 0x55efc408cf70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408e870 0x55efc409f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 --2- 172.21.15.123:0/3779693171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc408de80 0x55efc408ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 -- 172.21.15.123:0/3779693171 >> 172.21.15.123:0/3779693171 conn(0x55efc4056200 msgr2=0x55efc408fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.863+0000 7f23ca393740 1 -- 172.21.15.123:0/3779693171 shutdown_connections 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.864+0000 7f23ca393740 1 -- 172.21.15.123:0/3779693171 wait complete. 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.864+0000 7f23ca393740 1 -- start start 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.864+0000 7f23ca393740 1 Processor -- start 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc409c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b2090 con 0x55efc40939e0 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b33b0 con 0x55efc408de80 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01b26f0 con 0x55efc408e870 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23ca393740 10 client.0 _ll_register_callbacks cb 0x7f23b000af00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23c5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57612/0 (socket says 172.21.15.123:57612) 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 -- 172.21.15.123:0/439284872 learned_addr learned my addr 172.21.15.123:0/439284872 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23c4c00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc409c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 -- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 msgr2=0x55efc409c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc409c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 -- 172.21.15.123:0/439284872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 msgr2=0x55efc409dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.865+0000 7f23bfe00640 1 -- 172.21.15.123:0/439284872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efc409e200 con 0x55efc408e870 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.866+0000 7f23bfe00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f23b0004650 tx=0x7f23b0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd321a8df415ba04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.866+0000 7f23a2000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.866+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0036060 con 0x55efc408e870 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.866+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23b80bd930 con 0x55efc408e870 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.866+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f23b0009040 con 0x55efc408e870 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23b0017040 con 0x55efc408e870 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 10 client.26830 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55efc409f580 con 0x55efc408e870 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b80bd930 con 0x55efc408e870 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f23b000b040 con 0x55efc408e870 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 10 client.26830 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 10 client.26830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 20 client.26830 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40ff770 0x55efc4101b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 4 client.26830 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.867+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55efc3ffaae0 con 0x55efc40ff770 2024-03-24T00:49:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.868+0000 7f23c4c00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40ff770 0x55efc4101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.868+0000 7f23a2000640 10 client.26830 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.868+0000 7f23c4c00640 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40ff770 0x55efc4101b90 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1b6b4be3d513295 server_cookie=85f2bc0fee68e191 in_seq=0 out_seq=0 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23a2000640 1 -- 172.21.15.123:0/439284872 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55efc3ffaae0 con 0x55efc40ff770 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23a2000640 10 client.26830 handle_command_reply: tid=0 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 client.26830 shutdown 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 msgr2=0x55efc409cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f23b0004650 tx=0x7f23b0004730 comp rx=0 tx=0).stop 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 shutdown_connections 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55efc40939e0 0x55efc409dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55efc40ff770 0x55efc4101b90 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55efc408e870 0x55efc409cc00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 --2- 172.21.15.123:0/439284872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efc408de80 0x55efc409c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23a2000640 0 client.26830 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 >> 172.21.15.123:0/439284872 conn(0x55efc4056200 msgr2=0x55efc408f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 shutdown_connections 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 1 -- 172.21.15.123:0/439284872 wait complete. 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.869+0000 7f23ca393740 20 client.26830 trim_cache size 0 max 16384 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:49:31.873 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 msgr2=0x7f23a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 0x7f23a80a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f23b000afd0 tx=0x7f23b0005f90 comp rx=0 tx=0).stop 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 msgr2=0x7f23c010b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f23b802c640 tx=0x7f23b8035d10 comp rx=0 tx=0).stop 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 shutdown_connections 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010a0f0 0x7f23c010b140 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f23a80a2880 0x7f23a80a4d40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23c0109750 0x7f23c010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 --2- 172.21.15.123:0/3836217133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23c0108a90 0x7f23c010c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:31.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.870+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 >> 172.21.15.123:0/3836217133 conn(0x7f23c006e8a0 msgr2=0x7f23c010ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:31.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.871+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 shutdown_connections 2024-03-24T00:49:31.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:31.871+0000 7f23c7600640 1 -- 172.21.15.123:0/3836217133 wait complete. 2024-03-24T00:49:31.889 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:49:31.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:32.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 -- 172.21.15.123:0/227222585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d88102160 msgr2=0x7f3d88104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 --2- 172.21.15.123:0/227222585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d88102160 0x7f3d88104550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009900 tx=0x7f3d7802bc40 comp rx=0 tx=0).stop 2024-03-24T00:49:32.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 -- 172.21.15.123:0/227222585 shutdown_connections 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 --2- 172.21.15.123:0/227222585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d881073c0 0x7f3d88109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 --2- 172.21.15.123:0/227222585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88104a90 0x7f3d88106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 --2- 172.21.15.123:0/227222585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d88102160 0x7f3d88104550 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 -- 172.21.15.123:0/227222585 >> 172.21.15.123:0/227222585 conn(0x7f3d880fbb50 msgr2=0x7f3d880fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 -- 172.21.15.123:0/227222585 shutdown_connections 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.079+0000 7f3d8e200640 1 -- 172.21.15.123:0/227222585 wait complete. 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.080+0000 7f3d8e200640 1 Processor -- start 2024-03-24T00:49:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.080+0000 7f3d8e200640 1 -- start start 2024-03-24T00:49:32.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 0x7f3d88101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 0x7f3d88100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8810d3b0 con 0x7f3d88104a90 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8810d230 con 0x7f3d88102160 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d8e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8810d530 con 0x7f3d881073c0 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d86a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d87400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 0x7f3d88101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 0x7f3d88100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d86a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58172/0 (socket says 172.21.15.123:58172) 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.081+0000 7f3d86a00640 1 -- 172.21.15.123:0/2910558826 learned_addr learned my addr 172.21.15.123:0/2910558826 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 -- 172.21.15.123:0/2910558826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 msgr2=0x7f3d88100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 0x7f3d88100650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 -- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 msgr2=0x7f3d88100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 -- 172.21.15.123:0/2910558826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88100f10 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87e00640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 0x7f3d88100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d86a00640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d87400640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 0x7f3d88101a80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d78036040 tx=0x7f3d7802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78041070 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d881a1b10 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d78004450 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78003e60 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.082+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d881a1fd0 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.084+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d880fd640 con 0x7f3d88102160 2024-03-24T00:49:32.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.085+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3d78044070 con 0x7f3d88102160 2024-03-24T00:49:32.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.086+0000 7f3d7fe00640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.087+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d780f0520 con 0x7f3d88102160 2024-03-24T00:49:32.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.087+0000 7f3d86a00640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.087+0000 7f3d86a00640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3d881017f0 tx=0x7f3d74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.091+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d780bcf20 con 0x7f3d88102160 2024-03-24T00:49:32.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.332+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8802d070 con 0x7f3d88102160 2024-03-24T00:49:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.333+0000 7f3d7fe00640 1 -- 172.21.15.123:0/2910558826 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7f3d780c0fc0 con 0x7f3d88102160 2024-03-24T00:49:32.333 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:49:32.333 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:49:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:49:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 msgr2=0x7f3d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3d881017f0 tx=0x7f3d74005f90 comp rx=0 tx=0).stop 2024-03-24T00:49:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 msgr2=0x7f3d88101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 0x7f3d88101a80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d78036040 tx=0x7f3d7802c660 comp rx=0 tx=0).stop 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 shutdown_connections 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d881073c0 0x7f3d88100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d88104a90 0x7f3d88100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.335+0000 7f3d8e200640 1 --2- 172.21.15.123:0/2910558826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d88102160 0x7f3d88101a80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.336+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 >> 172.21.15.123:0/2910558826 conn(0x7f3d880fbb50 msgr2=0x7f3d88102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.336+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 shutdown_connections 2024-03-24T00:49:32.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.336+0000 7f3d8e200640 1 -- 172.21.15.123:0/2910558826 wait complete. 2024-03-24T00:49:32.355 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:49:32.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:49:32.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 -- 172.21.15.123:0/427503584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981049b0 msgr2=0x7f9e98104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 --2- 172.21.15.123:0/427503584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981049b0 0x7f9e98104e30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009900 tx=0x7f9e8802bc30 comp rx=0 tx=0).stop 2024-03-24T00:49:32.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 -- 172.21.15.123:0/427503584 shutdown_connections 2024-03-24T00:49:32.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 --2- 172.21.15.123:0/427503584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e98105370 0x7f9e9810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 --2- 172.21.15.123:0/427503584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981049b0 0x7f9e98104e30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 --2- 172.21.15.123:0/427503584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e981037c0 0x7f9e98103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 -- 172.21.15.123:0/427503584 >> 172.21.15.123:0/427503584 conn(0x7f9e980fed50 msgr2=0x7f9e98101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 -- 172.21.15.123:0/427503584 shutdown_connections 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.545+0000 7f9e9f200640 1 -- 172.21.15.123:0/427503584 wait complete. 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.546+0000 7f9e9f200640 1 Processor -- start 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.546+0000 7f9e9f200640 1 -- start start 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 0x7f9e98199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98105370 0x7f9e981a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f470 con 0x7f9e981049b0 2024-03-24T00:49:32.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f2f0 con 0x7f9e981037c0 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9810f5f0 con 0x7f9e98105370 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 0x7f9e98199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33014/0 (socket says 172.21.15.123:33014) 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9de00640 1 -- 172.21.15.123:0/1648505184 learned_addr learned my addr 172.21.15.123:0/1648505184 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98105370 0x7f9e981a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 -- 172.21.15.123:0/1648505184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98105370 msgr2=0x7f9e981a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98105370 0x7f9e981a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 -- 172.21.15.123:0/1648505184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 msgr2=0x7f9e98199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9d400640 1 -- 172.21.15.123:0/1648505184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e981a16b0 con 0x7f9e981049b0 2024-03-24T00:49:32.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.547+0000 7f9e9de00640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 0x7f9e98199fd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e88036040 tx=0x7f9e8802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e92a00640 1 -- 172.21.15.123:0/1648505184 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88043020 con 0x7f9e981049b0 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e92a00640 1 -- 172.21.15.123:0/1648505184 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e880054e0 con 0x7f9e981049b0 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e981a1940 con 0x7f9e981049b0 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e92a00640 1 -- 172.21.15.123:0/1648505184 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88003ca0 con 0x7f9e981049b0 2024-03-24T00:49:32.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.548+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e981a1e60 con 0x7f9e981049b0 2024-03-24T00:49:32.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.551+0000 7f9e92a00640 1 -- 172.21.15.123:0/1648505184 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9e88046020 con 0x7f9e981049b0 2024-03-24T00:49:32.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.551+0000 7f9ea35e8740 1 Processor -- start 2024-03-24T00:49:32.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 -- start start 2024-03-24T00:49:32.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed796fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed7977e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed78d42c0 con 0x560ed796f9f0 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed787cc90 con 0x560ed7971580 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed78d0360 con 0x560ed7970bc0 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed796fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed7977e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57644/0 (socket says 172.21.15.123:57644) 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9de00640 1 -- 172.21.15.123:0/3973833550 learned_addr learned my addr 172.21.15.123:0/3973833550 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.552+0000 7f9e9de00640 1 -- 172.21.15.123:0/3973833550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 msgr2=0x560ed7977e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed7977e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9de00640 1 -- 172.21.15.123:0/3973833550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 msgr2=0x560ed796fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed796fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9de00640 1 -- 172.21.15.123:0/3973833550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ed78a1080 con 0x560ed7970bc0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c010770 tx=0x7f9e8c010c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c0c4883c4c2336 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e92a00640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 0x7f9e800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e87000640 1 -- 172.21.15.123:0/3973833550 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00e930 con 0x560ed7970bc0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e92a00640 1 -- 172.21.15.123:0/1648505184 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9e880f0b10 con 0x7f9e981049b0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e87000640 1 -- 172.21.15.123:0/3973833550 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c011440 con 0x560ed7970bc0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e87000640 1 -- 172.21.15.123:0/3973833550 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c019630 con 0x560ed7970bc0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.553+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 0x7f9e800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3973833550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 msgr2=0x560ed7971040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c010770 tx=0x7f9e8c010c30 comp rx=0 tx=0).stop 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 0x7f9e800a4cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x560ed7970ac0 tx=0x7f9e94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3973833550 shutdown_connections 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed7977e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7970bc0 0x560ed7971040 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3973833550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed796fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3973833550 >> 172.21.15.123:0/3973833550 conn(0x560ed796ade0 msgr2=0x560ed796b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3973833550 shutdown_connections 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3973833550 wait complete. 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.554+0000 7f9ea35e8740 1 -- start start 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 Processor -- start 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7979c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed79811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed7978590 con 0x560ed7970bc0 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed78900d0 con 0x560ed7971580 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ed7978710 con 0x560ed796f9f0 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9ea35e8740 10 client.0 _ll_register_callbacks cb 0x560ed7894e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7979c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58226/0 (socket says 172.21.15.123:58226) 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9e9d400640 1 -- 172.21.15.123:0/1655691685 learned_addr learned my addr 172.21.15.123:0/1655691685 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.555+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed79811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 -- 172.21.15.123:0/1655691685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 msgr2=0x560ed7979c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7979c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 -- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 msgr2=0x560ed79811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed79811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 -- 172.21.15.123:0/1655691685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ed75e83f0 con 0x560ed7970bc0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9e880bd160 tx=0x7f9e88005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d7bc97c3e1b2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e7b400640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880bd230 con 0x560ed7970bc0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9e880c5770 con 0x560ed7970bc0 2024-03-24T00:49:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e8c011440 con 0x560ed7970bc0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e880bcdc0 con 0x560ed7970bc0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9ea35e8740 10 client.16668 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.556+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560ed76b27d0 con 0x560ed7970bc0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c011440 con 0x560ed7970bc0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9e880d6090 con 0x560ed7970bc0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 10 client.16668 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 20 client.16668 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed7985680 0x560ed7987ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 4 client.16668 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ed78e1800 con 0x560ed7985680 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.557+0000 7f9e9de00640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed7985680 0x560ed7987ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9e9de00640 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed7985680 0x560ed7987ac0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7e6b1ed75bf651a server_cookie=8845497cd8a44acf in_seq=0 out_seq=0 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9e7b400640 10 client.16668 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:32.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9e7b400640 1 -- 172.21.15.123:0/1655691685 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560ed78e1800 con 0x560ed7985680 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9e7b400640 10 client.16668 handle_command_reply: tid=0 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9ea35e8740 1 client.16668 shutdown 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 msgr2=0x560ed797a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.558+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9e880bd160 tx=0x7f9e88005bd0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 shutdown_connections 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed79811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed797a1a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed7985680 0x560ed7987ac0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1655691685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7979c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 >> 172.21.15.123:0/1655691685 conn(0x560ed796ade0 msgr2=0x560ed796a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9e7b400640 0 client.16668 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 shutdown_connections 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1655691685 wait complete. 2024-03-24T00:49:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.559+0000 7f9ea35e8740 20 client.16668 trim_cache size 0 max 16384 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 Processor -- start 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 -- start start 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7969fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed79811a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981a8b00 con 0x560ed7970bc0 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981ab0c0 con 0x560ed7971580 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981a9100 con 0x560ed796f9f0 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed79811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9e9e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9e9e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33024/0 (socket says 172.21.15.123:33024) 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9e9e800640 1 -- 172.21.15.123:0/3792418829 learned_addr learned my addr 172.21.15.123:0/3792418829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.573+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7969fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 -- 172.21.15.123:0/3792418829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 msgr2=0x560ed798c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 -- 172.21.15.123:0/3792418829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 msgr2=0x560ed79811a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed79811a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 -- 172.21.15.123:0/3792418829 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ed7979080 con 0x560ed796f9f0 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9e800640 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e9de00640 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7969fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b770 tx=0x7f9e8c00b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa3e83d7339ed70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e86600640 1 -- 172.21.15.123:0/3792418829 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00bb90 con 0x560ed796f9f0 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e86600640 1 -- 172.21.15.123:0/3792418829 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c02a070 con 0x560ed796f9f0 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.574+0000 7f9e86600640 1 -- 172.21.15.123:0/3792418829 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00ecf0 con 0x560ed796f9f0 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3792418829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 msgr2=0x560ed7969fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7969fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b770 tx=0x7f9e8c00b4a0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3792418829 shutdown_connections 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7971580 0x560ed798c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed7970bc0 0x560ed79811a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/3792418829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed796f9f0 0x560ed7969fe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3792418829 >> 172.21.15.123:0/3792418829 conn(0x560ed796ade0 msgr2=0x560ed79698d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3792418829 shutdown_connections 2024-03-24T00:49:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- 172.21.15.123:0/3792418829 wait complete. 2024-03-24T00:49:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.575+0000 7f9ea35e8740 1 -- start start 2024-03-24T00:49:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 Processor -- start 2024-03-24T00:49:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed7980aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 0x560ed797d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981a9740 con 0x560ed796f9f0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981aaa60 con 0x560ed7970bc0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981a9da0 con 0x560ed7971580 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9ea35e8740 10 client.0 _ll_register_callbacks cb 0x7f9e88009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33032/0 (socket says 172.21.15.123:33032) 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9e9d400640 1 -- 172.21.15.123:0/1875605954 learned_addr learned my addr 172.21.15.123:0/1875605954 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed7980aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.576+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 0x560ed797d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 -- 172.21.15.123:0/1875605954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 msgr2=0x560ed797d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 0x560ed797d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 -- 172.21.15.123:0/1875605954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 msgr2=0x560ed7980aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed7980aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 -- 172.21.15.123:0/1875605954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ed797dc50 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9e800640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 0x560ed797d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e9d400640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9e880c15b0 tx=0x7f9e880037d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58fbe97b214d1213 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e7b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880bcfa0 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9e880bc8b0 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e8c02a070 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e880bd230 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9ea35e8740 10 client.26855 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560ed79e1c20 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.577+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c02a070 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9e880b91b0 con 0x560ed7970bc0 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 10 client.26855 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:32.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 20 client.26855 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed79e23f0 0x560ed79e4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 4 client.26855 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560ed78dbb50 con 0x560ed79e23f0 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e9de00640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed79e23f0 0x560ed79e4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e9de00640 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed79e23f0 0x560ed79e4810 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17a85eabff1dbba3 server_cookie=54d38fed47cf80c3 in_seq=0 out_seq=0 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e7b400640 10 client.26855 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e7b400640 1 -- 172.21.15.123:0/1875605954 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560ed78dbb50 con 0x560ed79e23f0 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9e7b400640 10 client.26855 handle_command_reply: tid=0 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.578+0000 7f9ea35e8740 1 client.26855 shutdown 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 msgr2=0x560ed7980fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9e880c15b0 tx=0x7f9e880037d0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 shutdown_connections 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x560ed79e23f0 0x560ed79e4810 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560ed7971580 0x560ed797d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ed7970bc0 0x560ed7980fe0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 --2- 172.21.15.123:0/1875605954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560ed796f9f0 0x560ed7980aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9e7b400640 0 client.26855 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 >> 172.21.15.123:0/1875605954 conn(0x560ed796ade0 msgr2=0x560ed796a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 shutdown_connections 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 1 -- 172.21.15.123:0/1875605954 wait complete. 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9ea35e8740 20 client.26855 trim_cache size 0 max 16384 2024-03-24T00:49:32.591 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 msgr2=0x7f9e800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 0x7f9e800a4cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x560ed7970ac0 tx=0x7f9e94009500 comp rx=0 tx=0).stop 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 msgr2=0x7f9e98199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.579+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 0x7f9e98199fd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e88036040 tx=0x7f9e8802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 shutdown_connections 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98105370 0x7f9e981a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f9e800a2800 0x7f9e800a4cc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e981049b0 0x7f9e98199fd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 --2- 172.21.15.123:0/1648505184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e981037c0 0x7f9e98199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 >> 172.21.15.123:0/1648505184 conn(0x7f9e980fed50 msgr2=0x7f9e98107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 shutdown_connections 2024-03-24T00:49:32.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:32.580+0000 7f9e9f200640 1 -- 172.21.15.123:0/1648505184 wait complete. 2024-03-24T00:49:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:32 smithi123 ceph-mon[31487]: pgmap v734: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-03-24T00:49:32.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:32 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2910558826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:49:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:32 smithi204 ceph-mon[36345]: pgmap v734: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-03-24T00:49:32.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:32 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2910558826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:49:32.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:32 smithi177 ceph-mon[36034]: pgmap v734: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-03-24T00:49:32.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:32 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2910558826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:49:33.597 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:49:33.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:33.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:49:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 -- 172.21.15.123:0/306142402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00076060 msgr2=0x7feb00076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 --2- 172.21.15.123:0/306142402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00076060 0x7feb00076480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009900 tx=0x7feaf002bc40 comp rx=0 tx=0).stop 2024-03-24T00:49:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 -- 172.21.15.123:0/306142402 shutdown_connections 2024-03-24T00:49:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 --2- 172.21.15.123:0/306142402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00077c40 0x7feb000784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 --2- 172.21.15.123:0/306142402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb000772a0 0x7feb00077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 --2- 172.21.15.123:0/306142402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00076060 0x7feb00076480 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 -- 172.21.15.123:0/306142402 >> 172.21.15.123:0/306142402 conn(0x7feb000fee50 msgr2=0x7feb00101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 -- 172.21.15.123:0/306142402 shutdown_connections 2024-03-24T00:49:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.796+0000 7feb06400640 1 -- 172.21.15.123:0/306142402 wait complete. 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.797+0000 7feb06400640 1 Processor -- start 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.797+0000 7feb06400640 1 -- start start 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.797+0000 7feb06400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.797+0000 7feb06400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feb06400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb00077c40 0x7feb001a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feb06400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001139b0 con 0x7feb00076060 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feb06400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb00113830 con 0x7feb000772a0 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feb06400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb00113b30 con 0x7feb00077c40 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaff400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaff400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58266/0 (socket says 172.21.15.123:58266) 2024-03-24T00:49:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaff400640 1 -- 172.21.15.123:0/1227635509 learned_addr learned my addr 172.21.15.123:0/1227635509 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33056/0 (socket says 172.21.15.123:33056) 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaffe00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb00077c40 0x7feb001a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 -- 172.21.15.123:0/1227635509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb00077c40 msgr2=0x7feb001a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb00077c40 0x7feb001a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 -- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 msgr2=0x7feb0019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaf7e00640 1 -- 172.21.15.123:0/1227635509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb001a5be0 con 0x7feb000772a0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.798+0000 7feaff400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7feae800e9f0 tx=0x7feae800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feafcc00640 1 -- 172.21.15.123:0/1227635509 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae800cd80 con 0x7feb000772a0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feafcc00640 1 -- 172.21.15.123:0/1227635509 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feae80035f0 con 0x7feb000772a0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb001a5ed0 con 0x7feb000772a0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feafcc00640 1 -- 172.21.15.123:0/1227635509 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8010690 con 0x7feb000772a0 2024-03-24T00:49:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.799+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb001a63e0 con 0x7feb000772a0 2024-03-24T00:49:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.802+0000 7feb0907e740 1 Processor -- start 2024-03-24T00:49:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.802+0000 7feafcc00640 1 -- 172.21.15.123:0/1227635509 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feae8003b30 con 0x7feb000772a0 2024-03-24T00:49:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.802+0000 7feb0907e740 1 -- start start 2024-03-24T00:49:33.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 0x55983e2e9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e246eb0 con 0x55983e2e10a0 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e1efa60 con 0x55983e2e6c00 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feb0907e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e242f50 con 0x55983e2e1a90 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 0x55983e2e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57718/0 (socket says 172.21.15.123:57718) 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 -- 172.21.15.123:0/3835351267 learned_addr learned my addr 172.21.15.123:0/3835351267 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaffe00640 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 -- 172.21.15.123:0/3835351267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 msgr2=0x55983e2e9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 0x55983e2e9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 -- 172.21.15.123:0/3835351267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 msgr2=0x55983e2e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaff400640 1 -- 172.21.15.123:0/3835351267 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55983e0237e0 con 0x55983e2e1a90 2024-03-24T00:49:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.803+0000 7feaf7e00640 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 0x55983e2e9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feaff400640 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feaf002c180 tx=0x7feaf0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47f5de291dc28343 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feaf4c00640 1 -- 172.21.15.123:0/3835351267 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0042070 con 0x55983e2e1a90 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feafcc00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 0x7feae40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feaf4c00640 1 -- 172.21.15.123:0/3835351267 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf002cd40 con 0x55983e2e1a90 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feaf4c00640 1 -- 172.21.15.123:0/3835351267 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0005370 con 0x55983e2e1a90 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feafcc00640 1 -- 172.21.15.123:0/1227635509 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feae80cec80 con 0x7feb000772a0 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.804+0000 7feaffe00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 0x7feae40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 -- 172.21.15.123:0/3835351267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 msgr2=0x55983e2e6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feaf002c180 tx=0x7feaf0000bc0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 -- 172.21.15.123:0/3835351267 shutdown_connections 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feaffe00640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 0x7feae40a4d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55983e2e22f0 tx=0x7feaf80096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e6c00 0x55983e2e9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e6590 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 --2- 172.21.15.123:0/3835351267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 -- 172.21.15.123:0/3835351267 >> 172.21.15.123:0/3835351267 conn(0x55983e2dc610 msgr2=0x55983e2dca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 -- 172.21.15.123:0/3835351267 shutdown_connections 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.805+0000 7feb0907e740 1 -- 172.21.15.123:0/3835351267 wait complete. 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 -- start start 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 Processor -- start 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e10a0 0x55983e2eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e6c00 0x55983e2f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e2e9b50 con 0x55983e2e6c00 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e202eb0 con 0x55983e2e1a90 2024-03-24T00:49:33.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x55983e2e9cd0 con 0x55983e2e10a0 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feb0907e740 10 client.0 _ll_register_callbacks cb 0x55983e1f7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.806+0000 7feaff400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e10a0 0x55983e2eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33064/0 (socket says 172.21.15.123:33064) 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 -- 172.21.15.123:0/1356449840 learned_addr learned my addr 172.21.15.123:0/1356449840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaffe00640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e6c00 0x55983e2f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 -- 172.21.15.123:0/1356449840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e10a0 msgr2=0x55983e2eb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e10a0 0x55983e2eb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 -- 172.21.15.123:0/1356449840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e6c00 msgr2=0x55983e2f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e6c00 0x55983e2f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 -- 172.21.15.123:0/1356449840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55983e083f70 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaf7e00640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feae8097a90 tx=0x7feae80978d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa585ba67498195 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaed600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae809af90 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feae8097ca0 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feaf002cd40 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.807+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feae8086070 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 10 client.26867 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55983df1b170 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf002cd40 con 0x55983e2e1a90 2024-03-24T00:49:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feae8084bc0 con 0x55983e2e1a90 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 10 client.26867 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 10 client.26867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 20 client.26867 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2f7070 0x55983e2f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 4 client.26867 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55983e2543f0 con 0x55983e2f7070 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.808+0000 7feaff400640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2f7070 0x55983e2f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.809+0000 7feaff400640 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2f7070 0x55983e2f94d0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9009d2b2a9aa9e63 server_cookie=8056d0f706091b96 in_seq=0 out_seq=0 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.809+0000 7feaed600640 10 client.26867 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.809+0000 7feaed600640 1 -- 172.21.15.123:0/1356449840 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55983e2543f0 con 0x55983e2f7070 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.809+0000 7feaed600640 10 client.26867 handle_command_reply: tid=0 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.809+0000 7feb0907e740 1 client.26867 shutdown 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 msgr2=0x55983e2eb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feae8097a90 tx=0x7feae80978d0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 shutdown_connections 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e6c00 0x55983e2f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2e1a90 0x55983e2eb7f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2f7070 0x55983e2f94d0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 --2- 172.21.15.123:0/1356449840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e10a0 0x55983e2eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 >> 172.21.15.123:0/1356449840 conn(0x55983e2dc610 msgr2=0x55983e2db830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feaed600640 0 client.26867 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 shutdown_connections 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 1 -- 172.21.15.123:0/1356449840 wait complete. 2024-03-24T00:49:33.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.810+0000 7feb0907e740 20 client.26867 trim_cache size 0 max 16384 2024-03-24T00:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.827+0000 7feb0907e740 1 Processor -- start 2024-03-24T00:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.827+0000 7feb0907e740 1 -- start start 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001ad060 con 0x55983e2e10a0 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001af620 con 0x55983e2dc610 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feb0907e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001ad660 con 0x55983e2e1a90 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33072/0 (socket says 172.21.15.123:33072) 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 -- 172.21.15.123:0/3900102531 learned_addr learned my addr 172.21.15.123:0/3900102531 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaffe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaf7e00640 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 -- 172.21.15.123:0/3900102531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 msgr2=0x55983e2e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 -- 172.21.15.123:0/3900102531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 msgr2=0x55983e2f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaff400640 1 -- 172.21.15.123:0/3900102531 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55983e2dd380 con 0x55983e2dc610 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.828+0000 7feaf7e00640 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feaff400640 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feaf0036040 tx=0x7feaf0040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ce976ead3590d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feaefe00640 1 -- 172.21.15.123:0/3900102531 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0042040 con 0x55983e2dc610 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feaefe00640 1 -- 172.21.15.123:0/3900102531 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf002cd40 con 0x55983e2dc610 2024-03-24T00:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feaefe00640 1 -- 172.21.15.123:0/3900102531 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0054310 con 0x55983e2dc610 2024-03-24T00:49:33.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 -- 172.21.15.123:0/3900102531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 msgr2=0x55983e2e2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feaf0036040 tx=0x7feaf0040040 comp rx=0 tx=0).stop 2024-03-24T00:49:33.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 -- 172.21.15.123:0/3900102531 shutdown_connections 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 --2- 172.21.15.123:0/3900102531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2e2530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 -- 172.21.15.123:0/3900102531 >> 172.21.15.123:0/3900102531 conn(0x55983e2e04a0 msgr2=0x55983e2ea990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.829+0000 7feb0907e740 1 -- 172.21.15.123:0/3900102531 shutdown_connections 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 -- 172.21.15.123:0/3900102531 wait complete. 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 -- start start 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 Processor -- start 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.830+0000 7feb0907e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feb0907e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001adca0 con 0x55983e2e10a0 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feb0907e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001aefc0 con 0x55983e2dc610 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feb0907e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001ae300 con 0x55983e2e1a90 2024-03-24T00:49:33.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feb0907e740 10 client.0 _ll_register_callbacks cb 0x7feae800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33076/0 (socket says 172.21.15.123:33076) 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 -- 172.21.15.123:0/157991469 learned_addr learned my addr 172.21.15.123:0/157991469 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaf7e00640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaffe00640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 -- 172.21.15.123:0/157991469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 msgr2=0x55983e2f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 -- 172.21.15.123:0/157991469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 msgr2=0x55983e2f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 -- 172.21.15.123:0/157991469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55983e2f1400 con 0x55983e2dc610 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaff400640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feaf002c180 tx=0x7feaf0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279e7c7b8d83da23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.831+0000 7feaed600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0042040 con 0x55983e2dc610 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feaf0004020 con 0x55983e2dc610 2024-03-24T00:49:33.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feaf002cd40 con 0x55983e2dc610 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feaf00542f0 con 0x55983e2dc610 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feb0907e740 10 client.26873 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55983e353690 con 0x55983e2dc610 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf002cd40 con 0x55983e2dc610 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feaf005c030 con 0x55983e2dc610 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feb0907e740 10 client.26873 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.832+0000 7feb0907e740 10 client.26873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feb0907e740 20 client.26873 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2e6c00 0x55983e355b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feb0907e740 4 client.26873 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55983e24e740 con 0x55983e2e6c00 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feaf7e00640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2e6c00 0x55983e355b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feaf7e00640 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2e6c00 0x55983e355b60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e04e8f130c9592 server_cookie=d20a387971dde78f in_seq=0 out_seq=0 2024-03-24T00:49:33.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.833+0000 7feaed600640 10 client.26873 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:33.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.834+0000 7feaed600640 1 -- 172.21.15.123:0/157991469 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55983e24e740 con 0x55983e2e6c00 2024-03-24T00:49:33.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feaed600640 10 client.26873 handle_command_reply: tid=0 2024-03-24T00:49:33.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 client.26873 shutdown 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 msgr2=0x55983e2f2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feaf002c180 tx=0x7feaf0009260 comp rx=0 tx=0).stop 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 shutdown_connections 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x55983e2e6c00 0x55983e355b60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55983e2e1a90 0x55983e2f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55983e2e10a0 0x55983e2f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 --2- 172.21.15.123:0/157991469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55983e2dc610 0x55983e2f2320 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 >> 172.21.15.123:0/157991469 conn(0x55983e2e04a0 msgr2=0x55983e2ea100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feaed600640 0 client.26873 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 shutdown_connections 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 1 -- 172.21.15.123:0/157991469 wait complete. 2024-03-24T00:49:33.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.835+0000 7feb0907e740 20 client.26873 trim_cache size 0 max 16384 2024-03-24T00:49:33.839 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:49:33.839 INFO:teuthology.orchestra.run.smithi123.stdout: "return_code": 0, 2024-03-24T00:49:33.839 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_tag": "f43c3188-ffef-4de3-8c33-32c16f8d02a6", 2024-03-24T00:49:33.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "asynchronous" 2024-03-24T00:49:33.839 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 msgr2=0x7feae40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 0x7feae40a4d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55983e2e22f0 tx=0x7feaf80096a0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 msgr2=0x7feb0019e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7feae800e9f0 tx=0x7feae800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 shutdown_connections 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb00077c40 0x7feb001a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000772a0 0x7feb0019e500 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7feae40a2880 0x7feae40a4d40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 --2- 172.21.15.123:0/1227635509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb00076060 0x7feb0019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 >> 172.21.15.123:0/1227635509 conn(0x7feb000fee50 msgr2=0x7feb00100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 shutdown_connections 2024-03-24T00:49:33.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:33.836+0000 7feb06400640 1 -- 172.21.15.123:0/1227635509 wait complete. 2024-03-24T00:49:33.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:49:33.855 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:49:34.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 -- 172.21.15.123:0/2918789829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc109430 msgr2=0x7f66cc10b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 --2- 172.21.15.123:0/2918789829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc109430 0x7f66cc10b8d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009ca0 tx=0x7f66c402c180 comp rx=0 tx=0).stop 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 -- 172.21.15.123:0/2918789829 shutdown_connections 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 --2- 172.21.15.123:0/2918789829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc109430 0x7f66cc10b8d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 --2- 172.21.15.123:0/2918789829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc0740a0 0x7f66cc074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 --2- 172.21.15.123:0/2918789829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc075680 0x7f66cc073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 -- 172.21.15.123:0/2918789829 >> 172.21.15.123:0/2918789829 conn(0x7f66cc0fcb10 msgr2=0x7f66cc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 -- 172.21.15.123:0/2918789829 shutdown_connections 2024-03-24T00:49:34.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.044+0000 7f66d0a00640 1 -- 172.21.15.123:0/2918789829 wait complete. 2024-03-24T00:49:34.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.045+0000 7f66d0a00640 1 Processor -- start 2024-03-24T00:49:34.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.045+0000 7f66d0a00640 1 -- start start 2024-03-24T00:49:34.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 0x7f66cc199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc109430 0x7f66cc1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc10f400 con 0x7f66cc109430 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc10f280 con 0x7f66cc0740a0 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66d0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc10f580 con 0x7f66cc075680 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33102/0 (socket says 172.21.15.123:33102) 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 -- 172.21.15.123:0/3923562378 learned_addr learned my addr 172.21.15.123:0/3923562378 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66cb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc109430 0x7f66cc1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66c3e00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 0x7f66cc199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 -- 172.21.15.123:0/3923562378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 msgr2=0x7f66cc199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 0x7f66cc199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 -- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc109430 msgr2=0x7f66cc1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc109430 0x7f66cc1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.046+0000 7f66ca800640 1 -- 172.21.15.123:0/3923562378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc1a1630 con 0x7f66cc0740a0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66c3e00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 0x7f66cc199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66ca800640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e9f0 tx=0x7f66bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66c3400640 1 -- 172.21.15.123:0/3923562378 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00cd80 con 0x7f66cc0740a0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc1a1920 con 0x7f66cc0740a0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66c3400640 1 -- 172.21.15.123:0/3923562378 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66bc010910 con 0x7f66cc0740a0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66c3400640 1 -- 172.21.15.123:0/3923562378 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0188f0 con 0x7f66cc0740a0 2024-03-24T00:49:34.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.047+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc1a1e70 con 0x7f66cc0740a0 2024-03-24T00:49:34.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.050+0000 7f66c3400640 1 -- 172.21.15.123:0/3923562378 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f66bc010ac0 con 0x7f66cc0740a0 2024-03-24T00:49:34.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.050+0000 7f66d4001740 1 Processor -- start 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 -- start start 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b21470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 0x564387b26540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387a86250 con 0x564387b21050 2024-03-24T00:49:34.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387a2ea30 con 0x564387b26bb0 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66d4001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387a822f0 con 0x564387b21a40 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66cb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b21470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66ca800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 0x564387b26540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66ca800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 0x564387b26540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:57780/0 (socket says 172.21.15.123:57780) 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66ca800640 1 -- 172.21.15.123:0/1685277472 learned_addr learned my addr 172.21.15.123:0/1685277472 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.051+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33108/0 (socket says 172.21.15.123:33108) 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 -- 172.21.15.123:0/1685277472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 msgr2=0x564387b26540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 0x564387b26540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 -- 172.21.15.123:0/1685277472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 msgr2=0x564387b21470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b21470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 -- 172.21.15.123:0/1685277472 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564387a19690 con 0x564387b26bb0 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66cb200640 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b21470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:34.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009fa0 tx=0x7f66b40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d437da2feccf0c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3400640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 0x7f66b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66bb400640 1 -- 172.21.15.123:0/1685277472 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4004030 con 0x564387b26bb0 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66bb400640 1 -- 172.21.15.123:0/1685277472 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b4005500 con 0x564387b26bb0 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66c3400640 1 -- 172.21.15.123:0/3923562378 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66bc0cee50 con 0x7f66cc0740a0 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66bb400640 1 -- 172.21.15.123:0/1685277472 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4004e10 con 0x564387b26bb0 2024-03-24T00:49:34.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.052+0000 7f66cb200640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 0x7f66b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 -- 172.21.15.123:0/1685277472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 msgr2=0x564387b29040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009fa0 tx=0x7f66b40118d0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66cb200640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 0x7f66b00a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66c402c6c0 tx=0x7f66c4032670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 -- 172.21.15.123:0/1685277472 shutdown_connections 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b29040 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21a40 0x564387b26540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 --2- 172.21.15.123:0/1685277472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b21470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 -- 172.21.15.123:0/1685277472 >> 172.21.15.123:0/1685277472 conn(0x564387ae9200 msgr2=0x564387b1c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 -- 172.21.15.123:0/1685277472 shutdown_connections 2024-03-24T00:49:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.053+0000 7f66d4001740 1 -- 172.21.15.123:0/1685277472 wait complete. 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 -- start start 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 Processor -- start 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b2b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21a40 0x564387b2b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.054+0000 7f66d4001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387b29b60 con 0x564387b21a40 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66d4001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387a41e80 con 0x564387b26bb0 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66d4001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x564387b29ce0 con 0x564387b21050 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66d4001740 10 client.0 _ll_register_callbacks cb 0x564387a54130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:34.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21a40 0x564387b2b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33114/0 (socket says 172.21.15.123:33114) 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 -- 172.21.15.123:0/1947622822 learned_addr learned my addr 172.21.15.123:0/1947622822 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66ca800640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b2b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 -- 172.21.15.123:0/1947622822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 msgr2=0x564387b2b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b2b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 -- 172.21.15.123:0/1947622822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21a40 msgr2=0x564387b2b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21a40 0x564387b2b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.055+0000 7f66cb200640 1 -- 172.21.15.123:0/1947622822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643878647d0 con 0x564387b26bb0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66cb200640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c523d48053dc06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66afe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4050020 con 0x564387b26bb0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66c4003fc0 con 0x564387b26bb0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66b4005500 con 0x564387b26bb0 2024-03-24T00:49:34.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66c4039040 con 0x564387b26bb0 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66d4001740 10 client.26885 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56438779a3f0 con 0x564387b26bb0 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.056+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b4005500 con 0x564387b26bb0 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f66c404ca40 con 0x564387b26bb0 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 10 client.26885 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 20 client.26885 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b36e30 0x564387b39270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 4 client.26885 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:34.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564387a93790 con 0x564387b36e30 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66ca800640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b36e30 0x564387b39270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66ca800640 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b36e30 0x564387b39270 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bba75e18e38acbe7 server_cookie=19c171d86f2d0e27 in_seq=0 out_seq=0 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.057+0000 7f66afe00640 10 client.26885 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.058+0000 7f66afe00640 1 -- 172.21.15.123:0/1947622822 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564387a93790 con 0x564387b36e30 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.058+0000 7f66afe00640 10 client.26885 handle_command_reply: tid=0 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.058+0000 7f66d4001740 1 client.26885 shutdown 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 msgr2=0x564387b325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c40095c0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 shutdown_connections 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b26bb0 0x564387b325a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21a40 0x564387b2b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b36e30 0x564387b39270 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 --2- 172.21.15.123:0/1947622822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b2b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 >> 172.21.15.123:0/1947622822 conn(0x564387ae9200 msgr2=0x564387b1bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66afe00640 0 client.26885 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 shutdown_connections 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 1 -- 172.21.15.123:0/1947622822 wait complete. 2024-03-24T00:49:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.059+0000 7f66d4001740 20 client.26885 trim_cache size 0 max 16384 2024-03-24T00:49:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.076+0000 7f66d4001740 1 Processor -- start 2024-03-24T00:49:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 -- start start 2024-03-24T00:49:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b17920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b26bb0 0x564387b1b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1a8a50 con 0x564387b26bb0 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1ab010 con 0x564387b21a40 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66d4001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1a9050 con 0x564387b21050 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33118/0 (socket says 172.21.15.123:33118) 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66c3e00640 1 -- 172.21.15.123:0/1283400695 learned_addr learned my addr 172.21.15.123:0/1283400695 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66cb200640 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b26bb0 0x564387b1b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.077+0000 7f66ca800640 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b17920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 -- 172.21.15.123:0/1283400695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 msgr2=0x564387b17920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b17920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 -- 172.21.15.123:0/1283400695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b26bb0 msgr2=0x564387b1b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b26bb0 0x564387b1b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 -- 172.21.15.123:0/1283400695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56438775a0f0 con 0x564387b21a40 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66c3e00640 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f66b40058c0 tx=0x7f66b4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a9df91a885ce2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66baa00640 1 -- 172.21.15.123:0/1283400695 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b401d040 con 0x564387b21a40 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66baa00640 1 -- 172.21.15.123:0/1283400695 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b4005580 con 0x564387b21a40 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.078+0000 7f66baa00640 1 -- 172.21.15.123:0/1283400695 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b400f740 con 0x564387b21a40 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- 172.21.15.123:0/1283400695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 msgr2=0x564387b32560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f66b40058c0 tx=0x7f66b4009310 comp rx=0 tx=0).stop 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- 172.21.15.123:0/1283400695 shutdown_connections 2024-03-24T00:49:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b26bb0 0x564387b1b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b32560 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 --2- 172.21.15.123:0/1283400695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b21050 0x564387b17920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- 172.21.15.123:0/1283400695 >> 172.21.15.123:0/1283400695 conn(0x564387ae9200 msgr2=0x564387b22d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- 172.21.15.123:0/1283400695 shutdown_connections 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- 172.21.15.123:0/1283400695 wait complete. 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.079+0000 7f66d4001740 1 -- start start 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 Processor -- start 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b26bb0 0x564387b31a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1a9690 con 0x564387b21050 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1aa9b0 con 0x564387b21a40 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1a9cf0 con 0x564387b26bb0 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66d4001740 10 client.0 _ll_register_callbacks cb 0x7f66c4044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:34.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66ca800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33122/0 (socket says 172.21.15.123:33122) 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66c3e00640 1 -- 172.21.15.123:0/945613521 learned_addr learned my addr 172.21.15.123:0/945613521 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.080+0000 7f66cb200640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b26bb0 0x564387b31a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 -- 172.21.15.123:0/945613521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b26bb0 msgr2=0x564387b31a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b26bb0 0x564387b31a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 -- 172.21.15.123:0/945613521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 msgr2=0x564387b304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b304d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 -- 172.21.15.123:0/945613521 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564387b32010 con 0x564387b21a40 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66c3e00640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f66b4014040 tx=0x7f66b4005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e810828687944e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66afe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4032060 con 0x564387b21a40 2024-03-24T00:49:34.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66b40035f0 con 0x564387b21a40 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.081+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66b4005580 con 0x564387b21a40 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66b401d040 con 0x564387b21a40 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 10 client.26891 fetch_fsmap learned FSMap version 19 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564387b92ed0 con 0x564387b21a40 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b4005580 con 0x564387b21a40 2024-03-24T00:49:34.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f66b400fc90 con 0x564387b21a40 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 10 client.26891 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 10 client.26891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 20 client.26891 populate_metadata read hostname 'smithi123' 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b93590 0x564387b959b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 4 client.26891 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:49:34.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564387a8dae0 con 0x564387b93590 2024-03-24T00:49:34.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.082+0000 7f66ca800640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b93590 0x564387b959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66ca800640 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b93590 0x564387b959b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36b64bb81971c69b server_cookie=630c57a9726e7fb1 in_seq=0 out_seq=0 2024-03-24T00:49:34.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66afe00640 10 client.26891 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:34.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66afe00640 1 -- 172.21.15.123:0/945613521 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564387a8dae0 con 0x564387b93590 2024-03-24T00:49:34.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66afe00640 10 client.26891 handle_command_reply: tid=0 2024-03-24T00:49:34.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 client.26891 shutdown 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 msgr2=0x564387b30a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f66b4014040 tx=0x7f66b4005b00 comp rx=0 tx=0).stop 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 shutdown_connections 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x564387b93590 0x564387b959b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564387b26bb0 0x564387b31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564387b21a40 0x564387b30a10 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 --2- 172.21.15.123:0/945613521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564387b21050 0x564387b304d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66afe00640 0 client.26891 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:49:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 >> 172.21.15.123:0/945613521 conn(0x564387ae9200 msgr2=0x564387b224e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 shutdown_connections 2024-03-24T00:49:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.083+0000 7f66d4001740 1 -- 172.21.15.123:0/945613521 wait complete. 2024-03-24T00:49:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d4001740 20 client.26891 trim_cache size 0 max 16384 2024-03-24T00:49:34.092 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:49:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-24T00:49:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": { 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "f43c3188-ffef-4de3-8c33-32c16f8d02a6": { 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/", 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "tag": "f43c3188-ffef-4de3-8c33-32c16f8d02a6", 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "options": "recursive,force" 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:49:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:49:34.094 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:49:34.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 msgr2=0x7f66b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 0x7f66b00a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66c402c6c0 tx=0x7f66c4032670 comp rx=0 tx=0).stop 2024-03-24T00:49:34.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 msgr2=0x7f66cc199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e9f0 tx=0x7f66bc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 shutdown_connections 2024-03-24T00:49:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f66b00a2800 0x7f66b00a4cc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc109430 0x7f66cc1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66cc075680 0x7f66cc199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 --2- 172.21.15.123:0/3923562378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66cc0740a0 0x7f66cc199a10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.084+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 >> 172.21.15.123:0/3923562378 conn(0x7f66cc0fcb10 msgr2=0x7f66cc1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.085+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 shutdown_connections 2024-03-24T00:49:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:49:34.085+0000 7f66d0a00640 1 -- 172.21.15.123:0/3923562378 wait complete. 2024-03-24T00:49:34.105 INFO:tasks.cephfs.filesystem:scrub status for tag:f43c3188-ffef-4de3-8c33-32c16f8d02a6 - {'path': '/', 'tag': 'f43c3188-ffef-4de3-8c33-32c16f8d02a6', 'options': 'recursive,force'} 2024-03-24T00:49:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:34 smithi204 ceph-mon[36345]: pgmap v735: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.1 KiB/s wr, 15 op/s 2024-03-24T00:49:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:34 smithi204 ceph-mon[36345]: scrub queued for path: / 2024-03-24T00:49:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:34 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:34 smithi204 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-24T00:49:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:34 smithi123 ceph-mon[31487]: pgmap v735: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.1 KiB/s wr, 15 op/s 2024-03-24T00:49:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:34 smithi123 ceph-mon[31487]: scrub queued for path: / 2024-03-24T00:49:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:34 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:34 smithi123 ceph-mon[31487]: scrub summary: active paths [/] 2024-03-24T00:49:34.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:34 smithi177 ceph-mon[36034]: pgmap v735: 97 pgs: 97 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.1 KiB/s wr, 15 op/s 2024-03-24T00:49:34.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:34 smithi177 ceph-mon[36034]: scrub queued for path: / 2024-03-24T00:49:34.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:34 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:34.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:34 smithi177 ceph-mon[36034]: scrub summary: active paths [/] 2024-03-24T00:49:35.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:35 smithi204 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:35.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:35 smithi123 ceph-mon[31487]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:35.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:35 smithi177 ceph-mon[36034]: scrub summary: idle+waiting paths [/] 2024-03-24T00:49:36.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:36 smithi204 ceph-mon[36345]: pgmap v736: 97 pgs: 97 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 22 op/s 2024-03-24T00:49:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:36 smithi123 ceph-mon[31487]: pgmap v736: 97 pgs: 97 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 22 op/s 2024-03-24T00:49:36.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:36 smithi177 ceph-mon[36034]: pgmap v736: 97 pgs: 97 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 22 op/s 2024-03-24T00:49:37.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:37 smithi204 ceph-mon[36345]: scrub summary: idle 2024-03-24T00:49:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:37 smithi123 ceph-mon[31487]: scrub summary: idle 2024-03-24T00:49:37.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:37 smithi177 ceph-mon[36034]: scrub summary: idle 2024-03-24T00:49:38.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:38 smithi204 ceph-mon[36345]: pgmap v737: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-03-24T00:49:38.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:38 smithi123 ceph-mon[31487]: pgmap v737: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-03-24T00:49:38.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:38 smithi177 ceph-mon[36034]: pgmap v737: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-03-24T00:49:38.990 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T00:49:38.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T00:49:40.361 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-24T00:49:40.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T00:49:40.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:40 smithi177 ceph-mon[36034]: pgmap v738: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.8 KiB/s wr, 26 op/s 2024-03-24T00:49:40.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:40 smithi204 ceph-mon[36345]: pgmap v738: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.8 KiB/s wr, 26 op/s 2024-03-24T00:49:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:40 smithi123 ceph-mon[31487]: pgmap v738: 97 pgs: 97 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.8 KiB/s wr, 26 op/s 2024-03-24T00:49:41.388 DEBUG:teuthology.parallel:result is None 2024-03-24T00:49:41.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T00:49:41.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:41 smithi204 ceph-mon[36345]: pgmap v739: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 49 KiB/s wr, 33 op/s 2024-03-24T00:49:41.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:41 smithi123 ceph-mon[31487]: pgmap v739: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 49 KiB/s wr, 33 op/s 2024-03-24T00:49:41.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:41 smithi177 ceph-mon[36034]: pgmap v739: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 49 KiB/s wr, 33 op/s 2024-03-24T00:49:44.622 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T00:49:44.622 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-24T00:49:44.633 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:49:44.659 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-24T00:49:44.669 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-24T00:49:44.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:44 smithi204 ceph-mon[36345]: pgmap v740: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T00:49:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:44 smithi123 ceph-mon[31487]: pgmap v740: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T00:49:44.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:44 smithi177 ceph-mon[36034]: pgmap v740: 97 pgs: 97 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T00:49:46.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:46 smithi204 ceph-mon[36345]: pgmap v741: 97 pgs: 97 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-03-24T00:49:46.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:46 smithi123 ceph-mon[31487]: pgmap v741: 97 pgs: 97 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-03-24T00:49:46.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:46 smithi177 ceph-mon[36034]: pgmap v741: 97 pgs: 97 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-03-24T00:49:48.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:48 smithi204 ceph-mon[36345]: pgmap v742: 97 pgs: 97 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 42 KiB/s wr, 23 op/s 2024-03-24T00:49:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:48 smithi123 ceph-mon[31487]: pgmap v742: 97 pgs: 97 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 42 KiB/s wr, 23 op/s 2024-03-24T00:49:48.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:48 smithi177 ceph-mon[36034]: pgmap v742: 97 pgs: 97 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 42 KiB/s wr, 23 op/s 2024-03-24T00:49:50.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:50 smithi204 ceph-mon[36345]: pgmap v743: 97 pgs: 97 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 21 op/s 2024-03-24T00:49:50.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:50 smithi123 ceph-mon[31487]: pgmap v743: 97 pgs: 97 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 21 op/s 2024-03-24T00:49:50.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:50 smithi177 ceph-mon[36034]: pgmap v743: 97 pgs: 97 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 21 op/s 2024-03-24T00:49:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:52 smithi204 ceph-mon[36345]: pgmap v744: 97 pgs: 97 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 KiB/s wr, 19 op/s 2024-03-24T00:49:52.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:52 smithi123 ceph-mon[31487]: pgmap v744: 97 pgs: 97 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 KiB/s wr, 19 op/s 2024-03-24T00:49:52.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:52 smithi177 ceph-mon[36034]: pgmap v744: 97 pgs: 97 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 KiB/s wr, 19 op/s 2024-03-24T00:49:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:54 smithi204 ceph-mon[36345]: pgmap v745: 97 pgs: 97 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 KiB/s wr, 10 op/s 2024-03-24T00:49:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:54 smithi123 ceph-mon[31487]: pgmap v745: 97 pgs: 97 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 KiB/s wr, 10 op/s 2024-03-24T00:49:54.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:54 smithi177 ceph-mon[36034]: pgmap v745: 97 pgs: 97 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 KiB/s wr, 10 op/s 2024-03-24T00:49:55.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:55 smithi204 ceph-mon[36345]: pgmap v746: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-03-24T00:49:55.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:55 smithi123 ceph-mon[31487]: pgmap v746: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-03-24T00:49:55.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:55 smithi177 ceph-mon[36034]: pgmap v746: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 10 KiB/s wr, 12 op/s 2024-03-24T00:49:58.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:49:58 smithi204 ceph-mon[36345]: pgmap v747: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.8 KiB/s wr, 10 op/s 2024-03-24T00:49:58.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:49:58 smithi123 ceph-mon[31487]: pgmap v747: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.8 KiB/s wr, 10 op/s 2024-03-24T00:49:58.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:49:58 smithi177 ceph-mon[36034]: pgmap v747: 97 pgs: 97 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.8 KiB/s wr, 10 op/s 2024-03-24T00:50:00.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:00 smithi204 ceph-mon[36345]: pgmap v748: 97 pgs: 97 active+clean; 8.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-03-24T00:50:00.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:00 smithi204 ceph-mon[36345]: overall HEALTH_OK 2024-03-24T00:50:00.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:00 smithi123 ceph-mon[31487]: pgmap v748: 97 pgs: 97 active+clean; 8.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-03-24T00:50:00.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:00 smithi123 ceph-mon[31487]: overall HEALTH_OK 2024-03-24T00:50:00.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:00 smithi177 ceph-mon[36034]: pgmap v748: 97 pgs: 97 active+clean; 8.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-03-24T00:50:00.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:00 smithi177 ceph-mon[36034]: overall HEALTH_OK 2024-03-24T00:50:01.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:01 smithi204 ceph-mon[36345]: pgmap v749: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-24T00:50:01.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:01 smithi123 ceph-mon[31487]: pgmap v749: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-24T00:50:01.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:01 smithi177 ceph-mon[36034]: pgmap v749: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-24T00:50:04.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:50:04.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 -- 172.21.15.123:0/315096959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c107390 msgr2=0x7f996c1097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 --2- 172.21.15.123:0/315096959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c107390 0x7f996c1097e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9964009ca0 tx=0x7f996402c100 comp rx=0 tx=0).stop 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 -- 172.21.15.123:0/315096959 shutdown_connections 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 --2- 172.21.15.123:0/315096959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c107390 0x7f996c1097e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 --2- 172.21.15.123:0/315096959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c104a60 0x7f996c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 --2- 172.21.15.123:0/315096959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c102130 0x7f996c104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 -- 172.21.15.123:0/315096959 >> 172.21.15.123:0/315096959 conn(0x7f996c0fbb40 msgr2=0x7f996c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.324+0000 7f9973600640 1 -- 172.21.15.123:0/315096959 shutdown_connections 2024-03-24T00:50:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.325+0000 7f9973600640 1 -- 172.21.15.123:0/315096959 wait complete. 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.325+0000 7f9973600640 1 Processor -- start 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 -- start start 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c104a60 0x7f996c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 0x7f996c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c10d330 con 0x7f996c104a60 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c10d1b0 con 0x7f996c107390 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9973600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c10d4b0 con 0x7f996c102130 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9970c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9970c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58214/0 (socket says 172.21.15.123:58214) 2024-03-24T00:50:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9970c00640 1 -- 172.21.15.123:0/3638348928 learned_addr learned my addr 172.21.15.123:0/3638348928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f9971600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 0x7f996c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.326+0000 7f996be00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c104a60 0x7f996c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 -- 172.21.15.123:0/3638348928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 msgr2=0x7f996c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 0x7f996c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 -- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c104a60 msgr2=0x7f996c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c104a60 0x7f996c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 -- 172.21.15.123:0/3638348928 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c19f5d0 con 0x7f996c102130 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9971600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 0x7f996c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9970c00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e9f0 tx=0x7f995c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9969600640 1 -- 172.21.15.123:0/3638348928 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c00cd80 con 0x7f996c102130 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9969600640 1 -- 172.21.15.123:0/3638348928 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f995c010910 con 0x7f996c102130 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9969600640 1 -- 172.21.15.123:0/3638348928 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c018990 con 0x7f996c102130 2024-03-24T00:50:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c19f8c0 con 0x7f996c102130 2024-03-24T00:50:04.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.327+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c19fe60 con 0x7f996c102130 2024-03-24T00:50:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.330+0000 7f99761df740 1 Processor -- start 2024-03-24T00:50:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 -- start start 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 0x561c04c5a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c5adf0 0x561c04c5d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04bb6250 con 0x561c04c5adf0 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04b5ea30 con 0x561c04c55c80 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f99761df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04bb22f0 con 0x561c04c55290 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f9969600640 1 -- 172.21.15.123:0/3638348928 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f995c010ac0 con 0x7f996c102130 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f9971600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f9971600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58220/0 (socket says 172.21.15.123:58220) 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f9971600640 1 -- 172.21.15.123:0/138252679 learned_addr learned my addr 172.21.15.123:0/138252679 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f9970c00640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 0x561c04c5a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.331+0000 7f996be00640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c5adf0 0x561c04c5d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 -- 172.21.15.123:0/138252679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 msgr2=0x561c04c5a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 0x561c04c5a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 -- 172.21.15.123:0/138252679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c5adf0 msgr2=0x561c04c5d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c5adf0 0x561c04c5d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 -- 172.21.15.123:0/138252679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c04b49690 con 0x561c04c55290 2024-03-24T00:50:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9970c00640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 0x561c04c5a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9971600640 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f996402c710 tx=0x7f99640323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a281f38b794a4f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9961c00640 1 -- 172.21.15.123:0/138252679 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964043070 con 0x561c04c55290 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9961c00640 1 -- 172.21.15.123:0/138252679 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9964032c90 con 0x561c04c55290 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.332+0000 7f9961c00640 1 -- 172.21.15.123:0/138252679 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964039040 con 0x561c04c55290 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f9969600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f9969600640 1 -- 172.21.15.123:0/3638348928 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f995c0ced00 con 0x7f996c102130 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f9970c00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 -- 172.21.15.123:0/138252679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 msgr2=0x561c04c556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f996402c710 tx=0x7f99640323d0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 -- 172.21.15.123:0/138252679 shutdown_connections 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c5adf0 0x561c04c5d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55c80 0x561c04c5a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 --2- 172.21.15.123:0/138252679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c556b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 -- 172.21.15.123:0/138252679 >> 172.21.15.123:0/138252679 conn(0x561c04c19200 msgr2=0x561c04c4c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 -- 172.21.15.123:0/138252679 shutdown_connections 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.333+0000 7f99761df740 1 -- 172.21.15.123:0/138252679 wait complete. 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.334+0000 7f9970c00640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 0x7f99500a4d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f995c0ccb80 tx=0x7f995c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.334+0000 7f99761df740 1 -- start start 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.334+0000 7f99761df740 1 Processor -- start 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c5adf0 0x561c04c66770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04c5dd40 con 0x561c04c55c80 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04b71e80 con 0x561c04c5adf0 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c04c5dec0 con 0x561c04c55290 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f99761df740 10 client.0 _ll_register_callbacks cb 0x561c04b84130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f996be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58234/0 (socket says 172.21.15.123:58234) 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 -- 172.21.15.123:0/3851094319 learned_addr learned my addr 172.21.15.123:0/3851094319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c5adf0 0x561c04c66770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 -- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c5adf0 msgr2=0x561c04c66770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c5adf0 0x561c04c66770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 -- 172.21.15.123:0/3851094319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 msgr2=0x561c04c5f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c5f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.335+0000 7f9971600640 1 -- 172.21.15.123:0/3851094319 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c049947d0 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f9971600640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9964009620 tx=0x7f996402e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd25ac6a7421257e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964039070 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9964003970 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9964032c90 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9964043040 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f99761df740 10 client.26902 fetch_fsmap learned FSMap version 19 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c048ca3f0 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9964032c90 con 0x561c04c55290 2024-03-24T00:50:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9964048450 con 0x561c04c55290 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f99761df740 10 client.26902 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.336+0000 7f99761df740 10 client.26902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f99761df740 20 client.26902 populate_metadata read hostname 'smithi123' 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04c6ac40 0x561c04c6d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f99761df740 4 client.26902 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c04bc3790 con 0x561c04c6ac40 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f996be00640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04c6ac40 0x561c04c6d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f996be00640 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04c6ac40 0x561c04c6d030 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e35f639b89bfd3 server_cookie=46cd6d84166d8361 in_seq=0 out_seq=0 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.337+0000 7f995a000640 10 client.26902 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f995a000640 1 -- 172.21.15.123:0/3851094319 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c04bc3790 con 0x561c04c6ac40 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f995a000640 10 client.26902 handle_command_reply: tid=0 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 client.26902 shutdown 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 msgr2=0x561c04c5f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9964009620 tx=0x7f996402e040 comp rx=0 tx=0).stop 2024-03-24T00:50:04.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 shutdown_connections 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c5adf0 0x561c04c66770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c5f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04c6ac40 0x561c04c6d030 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 --2- 172.21.15.123:0/3851094319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c55290 0x561c04c5f260 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f995a000640 0 client.26902 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.338+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 >> 172.21.15.123:0/3851094319 conn(0x561c04c19200 msgr2=0x561c04c4ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.339+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 shutdown_connections 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.339+0000 7f99761df740 1 -- 172.21.15.123:0/3851094319 wait complete. 2024-03-24T00:50:04.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.339+0000 7f99761df740 20 client.26902 trim_cache size 0 max 16384 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.361+0000 7f99761df740 1 Processor -- start 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 -- start start 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c66700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c4ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a4890 con 0x561c04c55c80 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a6e50 con 0x561c04c55290 2024-03-24T00:50:04.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f99761df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a4e90 con 0x561c04c5adf0 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f9971600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f996be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c66700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f9971600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33776/0 (socket says 172.21.15.123:33776) 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.362+0000 7f9971600640 1 -- 172.21.15.123:0/3756007826 learned_addr learned my addr 172.21.15.123:0/3756007826 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c4ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 -- 172.21.15.123:0/3756007826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 msgr2=0x561c04c4ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c4ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 -- 172.21.15.123:0/3756007826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 msgr2=0x561c04c66700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c66700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 -- 172.21.15.123:0/3756007826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c0488a0f0 con 0x561c04c55290 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9971600640 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f996402c640 tx=0x7f996404b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b1293a1751f510 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9961200640 1 -- 172.21.15.123:0/3756007826 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996405d020 con 0x561c04c55290 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9961200640 1 -- 172.21.15.123:0/3756007826 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9964032c90 con 0x561c04c55290 2024-03-24T00:50:04.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.363+0000 7f9961200640 1 -- 172.21.15.123:0/3756007826 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964039040 con 0x561c04c55290 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 -- 172.21.15.123:0/3756007826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 msgr2=0x561c04c4fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f996402c640 tx=0x7f996404b040 comp rx=0 tx=0).stop 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 -- 172.21.15.123:0/3756007826 shutdown_connections 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c4ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c66700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 --2- 172.21.15.123:0/3756007826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c4fa30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 -- 172.21.15.123:0/3756007826 >> 172.21.15.123:0/3756007826 conn(0x561c04c19200 msgr2=0x561c04c56fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 -- 172.21.15.123:0/3756007826 shutdown_connections 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.364+0000 7f99761df740 1 -- 172.21.15.123:0/3756007826 wait complete. 2024-03-24T00:50:04.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.365+0000 7f99761df740 1 -- start start 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.365+0000 7f99761df740 1 Processor -- start 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c64a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c65a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a54d0 con 0x561c04c55c80 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a67f0 con 0x561c04c55290 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a5b30 con 0x561c04c5adf0 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f99761df740 10 client.0 _ll_register_callbacks cb 0x7f9954005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f9971600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c64a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f9971600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33778/0 (socket says 172.21.15.123:33778) 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f9971600640 1 -- 172.21.15.123:0/3145059453 learned_addr learned my addr 172.21.15.123:0/3145059453 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f9971600640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c65a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 -- 172.21.15.123:0/3145059453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 msgr2=0x561c04c65a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c65a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 -- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 msgr2=0x561c04c64520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f996be00640 1 -- 172.21.15.123:0/3145059453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c04c66060 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.366+0000 7f9971600640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f996be00640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c64a60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x561c04c6d6c0 tx=0x7f9954004100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8bc73b5b5c7147 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f995a000640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954020d00 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9954005020 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9964032c90 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9954005220 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f99761df740 10 client.16728 fetch_fsmap learned FSMap version 19 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c04cc69f0 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.367+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9964032c90 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9954003790 con 0x561c04c55c80 2024-03-24T00:50:04.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 10 client.16728 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 10 client.16728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 20 client.16728 populate_metadata read hostname 'smithi123' 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04cc7370 0x561c04cc9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 4 client.16728 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561c04bbdae0 con 0x561c04cc7370 2024-03-24T00:50:04.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f9971600640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04cc7370 0x561c04cc9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f995a000640 10 client.16728 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.368+0000 7f9971600640 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04cc7370 0x561c04cc9790 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c72c1aed26f4f5 server_cookie=ca018a032e14cd90 in_seq=0 out_seq=0 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f995a000640 1 -- 172.21.15.123:0/3145059453 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561c04bbdae0 con 0x561c04cc7370 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f995a000640 10 client.16728 handle_command_reply: tid=0 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 client.16728 shutdown 2024-03-24T00:50:04.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 msgr2=0x561c04c64a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c64a60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x561c04c6d6c0 tx=0x7f9954004100 comp rx=0 tx=0).stop 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 shutdown_connections 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x561c04cc7370 0x561c04cc9790 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561c04c5adf0 0x561c04c65a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561c04c55c80 0x561c04c64a60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 --2- 172.21.15.123:0/3145059453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c04c55290 0x561c04c64520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f995a000640 0 client.16728 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 >> 172.21.15.123:0/3145059453 conn(0x561c04c19200 msgr2=0x561c04c4b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 shutdown_connections 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 1 -- 172.21.15.123:0/3145059453 wait complete. 2024-03-24T00:50:04.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.369+0000 7f99761df740 20 client.16728 trim_cache size 0 max 16384 2024-03-24T00:50:04.375 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:04.375 INFO:teuthology.orchestra.run.smithi123.stdout: "status": "no active scrubs running", 2024-03-24T00:50:04.375 INFO:teuthology.orchestra.run.smithi123.stdout: "scrubs": {} 2024-03-24T00:50:04.375 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 msgr2=0x7f99500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 0x7f99500a4d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f995c0ccb80 tx=0x7f995c000bc0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 msgr2=0x7f996c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e9f0 tx=0x7f995c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 shutdown_connections 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f996c107390 0x7f996c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f996c104a60 0x7f996c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 --2- 172.21.15.123:0/3638348928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996c102130 0x7f996c1979b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 >> 172.21.15.123:0/3638348928 conn(0x7f996c0fbb40 msgr2=0x7f996c102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 shutdown_connections 2024-03-24T00:50:04.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.370+0000 7f9973600640 1 -- 172.21.15.123:0/3638348928 wait complete. 2024-03-24T00:50:04.388 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:50:04.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:50:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 -- 172.21.15.123:0/539460337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 msgr2=0x7fa054106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 --2- 172.21.15.123:0/539460337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa054106e50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa040009900 tx=0x7fa04002bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 -- 172.21.15.123:0/539460337 shutdown_connections 2024-03-24T00:50:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 --2- 172.21.15.123:0/539460337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 0x7fa0541097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 --2- 172.21.15.123:0/539460337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa054106e50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 --2- 172.21.15.123:0/539460337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 -- 172.21.15.123:0/539460337 >> 172.21.15.123:0/539460337 conn(0x7fa0540fbb20 msgr2=0x7fa0540fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 -- 172.21.15.123:0/539460337 shutdown_connections 2024-03-24T00:50:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.580+0000 7fa05aa00640 1 -- 172.21.15.123:0/539460337 wait complete. 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 Processor -- start 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 -- start start 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa0541000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 0x7fa054100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05410d380 con 0x7fa054102130 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05410d200 con 0x7fa054104a60 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa05aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05410d500 con 0x7fa054107390 2024-03-24T00:50:04.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.581+0000 7fa053e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa0541000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44312/0 (socket says 172.21.15.123:44312) 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053e00640 1 -- 172.21.15.123:0/3137664629 learned_addr learned my addr 172.21.15.123:0/3137664629 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa058a00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 0x7fa054100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 -- 172.21.15.123:0/3137664629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 msgr2=0x7fa054100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 0x7fa054100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 -- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 msgr2=0x7fa054101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 -- 172.21.15.123:0/3137664629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054100eb0 con 0x7fa054104a60 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053e00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.582+0000 7fa053400640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa0541000e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa040009230 tx=0x7fa04002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.583+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040043020 con 0x7fa054104a60 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.583+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0400054e0 con 0x7fa054104a60 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.583+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040003b60 con 0x7fa054104a60 2024-03-24T00:50:04.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.583+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0541a1ac0 con 0x7fa054104a60 2024-03-24T00:50:04.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.583+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0541a1f90 con 0x7fa054104a60 2024-03-24T00:50:04.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.584+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0540688e0 con 0x7fa054104a60 2024-03-24T00:50:04.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.585+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa040046020 con 0x7fa054104a60 2024-03-24T00:50:04.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.587+0000 7fa050c00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 0x7fa0380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:04.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.587+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa0400f0a00 con 0x7fa054104a60 2024-03-24T00:50:04.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.587+0000 7fa053e00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 0x7fa0380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:04.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.588+0000 7fa053e00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 0x7fa0380a4d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa044011550 tx=0x7fa044005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:04.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.591+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa04004d020 con 0x7fa054104a60 2024-03-24T00:50:04.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:04 smithi204 ceph-mon[36345]: pgmap v750: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T00:50:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.836+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0540622a0 con 0x7fa054104a60 2024-03-24T00:50:04.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.836+0000 7fa050c00640 1 -- 172.21.15.123:0/3137664629 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fa0400bda50 con 0x7fa054104a60 2024-03-24T00:50:04.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:04.838 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:50:04.838 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:50:04.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 msgr2=0x7fa0380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 0x7fa0380a4d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa044011550 tx=0x7fa044005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 msgr2=0x7fa0541000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa0541000e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa040009230 tx=0x7fa04002c5d0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 shutdown_connections 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa054107390 0x7fa054100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa054104a60 0x7fa0541000e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fa0380a2880 0x7fa0380a4d40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.839+0000 7fa05aa00640 1 --2- 172.21.15.123:0/3137664629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa054102130 0x7fa054101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.840+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 >> 172.21.15.123:0/3137664629 conn(0x7fa0540fbb20 msgr2=0x7fa0541029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.840+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 shutdown_connections 2024-03-24T00:50:04.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:04.840+0000 7fa05aa00640 1 -- 172.21.15.123:0/3137664629 wait complete. 2024-03-24T00:50:04.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:04 smithi123 ceph-mon[31487]: pgmap v750: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T00:50:04.859 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:50:04.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:50:04.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:04 smithi177 ceph-mon[36034]: pgmap v750: 97 pgs: 97 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T00:50:05.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 -- 172.21.15.123:0/3841128768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714105490 msgr2=0x7fe71410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 --2- 172.21.15.123:0/3841128768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714105490 0x7fe71410bd20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009ca0 tx=0x7fe70c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 -- 172.21.15.123:0/3841128768 shutdown_connections 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 --2- 172.21.15.123:0/3841128768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714105490 0x7fe71410bd20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 --2- 172.21.15.123:0/3841128768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714104ad0 0x7fe714104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 --2- 172.21.15.123:0/3841128768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7141038e0 0x7fe714103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 -- 172.21.15.123:0/3841128768 >> 172.21.15.123:0/3841128768 conn(0x7fe7140fee50 msgr2=0x7fe714101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 -- 172.21.15.123:0/3841128768 shutdown_connections 2024-03-24T00:50:05.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.045+0000 7fe71ae00640 1 -- 172.21.15.123:0/3841128768 wait complete. 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 Processor -- start 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 -- start start 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe714104ad0 0x7fe71419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714105490 0x7fe7141a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71410f560 con 0x7fe714104ad0 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71410f3e0 con 0x7fe7141038e0 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.046+0000 7fe71ae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71410f6e0 con 0x7fe714105490 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe714104ad0 0x7fe71419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe718e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714105490 0x7fe7141a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33814/0 (socket says 172.21.15.123:33814) 2024-03-24T00:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 -- 172.21.15.123:0/54959467 learned_addr learned my addr 172.21.15.123:0/54959467 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 -- 172.21.15.123:0/54959467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714105490 msgr2=0x7fe7141a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714105490 0x7fe7141a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 -- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe714104ad0 msgr2=0x7fe71419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe714104ad0 0x7fe71419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 -- 172.21.15.123:0/54959467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7141a1790 con 0x7fe7141038e0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.047+0000 7fe713e00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe70400d950 tx=0x7fe70400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.048+0000 7fe710c00640 1 -- 172.21.15.123:0/54959467 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704017020 con 0x7fe7141038e0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.048+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7141a1a80 con 0x7fe7141038e0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.048+0000 7fe710c00640 1 -- 172.21.15.123:0/54959467 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe704009c30 con 0x7fe7141038e0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.048+0000 7fe710c00640 1 -- 172.21.15.123:0/54959467 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70400f720 con 0x7fe7141038e0 2024-03-24T00:50:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.048+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7141a1fd0 con 0x7fe7141038e0 2024-03-24T00:50:05.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 Processor -- start 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 -- start start 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe710c00640 1 -- 172.21.15.123:0/54959467 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe70401a020 con 0x7fe7141038e0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9d810 0x563308f9dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308f022c0 con 0x563308f9d810 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.052+0000 7fe71da1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308eaac90 con 0x563308f9e9e0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe71da1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308efe360 con 0x563308f9f3a0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe718e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9d810 0x563308f9dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33824/0 (socket says 172.21.15.123:33824) 2024-03-24T00:50:05.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 -- 172.21.15.123:0/692343525 learned_addr learned my addr 172.21.15.123:0/692343525 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:05.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 -- 172.21.15.123:0/692343525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 msgr2=0x563308fa5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 -- 172.21.15.123:0/692343525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9d810 msgr2=0x563308f9dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9d810 0x563308f9dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713e00640 1 -- 172.21.15.123:0/692343525 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563308ecf080 con 0x563308f9e9e0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.053+0000 7fe713400640 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe713e00640 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe7040d4b60 tx=0x7fe7040d6690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5192641afd8c1b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe709600640 1 -- 172.21.15.123:0/692343525 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7040dd070 con 0x563308f9e9e0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe710c00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 0x7fe6f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe709600640 1 -- 172.21.15.123:0/692343525 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7040d9b20 con 0x563308f9e9e0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe709600640 1 -- 172.21.15.123:0/692343525 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7040d8b20 con 0x563308f9e9e0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe710c00640 1 -- 172.21.15.123:0/54959467 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe7040ce960 con 0x7fe7141038e0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.054+0000 7fe718e00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 0x7fe6f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- 172.21.15.123:0/692343525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 msgr2=0x563308f9ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe7040d4b60 tx=0x7fe7040d6690 comp rx=0 tx=0).stop 2024-03-24T00:50:05.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- 172.21.15.123:0/692343525 shutdown_connections 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9e9e0 0x563308f9ee60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 --2- 172.21.15.123:0/692343525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9d810 0x563308f9dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- 172.21.15.123:0/692343525 >> 172.21.15.123:0/692343525 conn(0x563308f98c00 msgr2=0x563308f99000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe718e00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 0x7fe6f80a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe70c02c630 tx=0x7fe70c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- 172.21.15.123:0/692343525 shutdown_connections 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- 172.21.15.123:0/692343525 wait complete. 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.055+0000 7fe71da1d740 1 -- start start 2024-03-24T00:50:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 Processor -- start 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9d810 0x563308fa7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308fa8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308fa63b0 con 0x563308f9e9e0 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308ebe0d0 con 0x563308f9f3a0 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x563308fa6530 con 0x563308f9d810 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe71da1d740 10 client.0 _ll_register_callbacks cb 0x563308ec2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe718e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe718e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33830/0 (socket says 172.21.15.123:33830) 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe713e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9d810 0x563308fa7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe718e00640 1 -- 172.21.15.123:0/1548801330 learned_addr learned my addr 172.21.15.123:0/1548801330 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.056+0000 7fe713400640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308fa8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 -- 172.21.15.123:0/1548801330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9d810 msgr2=0x563308fa7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9d810 0x563308fa7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 -- 172.21.15.123:0/1548801330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 msgr2=0x563308fa8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308fa8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 -- 172.21.15.123:0/1548801330 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563308c163f0 con 0x563308f9f3a0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe718e00640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe70c006010 tx=0x7fe70c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600671a78e073ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe6fe000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c051020 con 0x563308f9f3a0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe70c04d680 con 0x563308f9f3a0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.057+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7040d9b20 con 0x563308f9f3a0 2024-03-24T00:50:05.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe70c039040 con 0x563308f9f3a0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 10 client.26927 fetch_fsmap learned FSMap version 19 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563308ce07d0 con 0x563308f9f3a0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7040d9b20 con 0x563308f9f3a0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe70c0349b0 con 0x563308f9f3a0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 10 client.26927 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 20 client.26927 populate_metadata read hostname 'smithi123' 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563308fb3550 0x563308fb5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 4 client.26927 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563308f0f800 con 0x563308fb3550 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.058+0000 7fe713e00640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563308fb3550 0x563308fb5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.059+0000 7fe713e00640 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563308fb3550 0x563308fb5990 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba294cd27242b87e server_cookie=866674ad475454d in_seq=0 out_seq=0 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.059+0000 7fe6fe000640 10 client.26927 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.059+0000 7fe6fe000640 1 -- 172.21.15.123:0/1548801330 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fe7040804b0 con 0x563308fb3550 2024-03-24T00:50:05.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.059+0000 7fe6fe000640 10 client.26927 handle_command_reply: tid=0 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 client.26927 shutdown 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 msgr2=0x563308faf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe70c006010 tx=0x7fe70c0095c0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 shutdown_connections 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9f3a0 0x563308faf0a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308fa8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563308fb3550 0x563308fb5990 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 --2- 172.21.15.123:0/1548801330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9d810 0x563308fa7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe6fe000640 0 client.26927 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 >> 172.21.15.123:0/1548801330 conn(0x563308f98c00 msgr2=0x563308f97eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 shutdown_connections 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 1 -- 172.21.15.123:0/1548801330 wait complete. 2024-03-24T00:50:05.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.060+0000 7fe71da1d740 20 client.26927 trim_cache size 0 max 16384 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 Processor -- start 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 -- start start 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308faf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141a8d70 con 0x563308f9e9e0 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141ab330 con 0x563308f9d810 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe71da1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141a9370 con 0x563308f9f3a0 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe718e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe713400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308faf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33846/0 (socket says 172.21.15.123:33846) 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe713e00640 1 -- 172.21.15.123:0/614377323 learned_addr learned my addr 172.21.15.123:0/614377323 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.074+0000 7fe718e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:58308/0 (socket says 172.21.15.123:58308) 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 -- 172.21.15.123:0/614377323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 msgr2=0x563308fa1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 -- 172.21.15.123:0/614377323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 msgr2=0x563308faf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308faf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 -- 172.21.15.123:0/614377323 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563308f94510 con 0x563308f9d810 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe718e00640 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe713e00640 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe704097ce0 tx=0x7fe704010e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877cb17f78e1e3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe708c00640 1 -- 172.21.15.123:0/614377323 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7040dd040 con 0x563308f9d810 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe708c00640 1 -- 172.21.15.123:0/614377323 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563308f0f800 con 0x563308f9d810 2024-03-24T00:50:05.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.075+0000 7fe708c00640 1 -- 172.21.15.123:0/614377323 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70409ba10 con 0x563308f9d810 2024-03-24T00:50:05.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- 172.21.15.123:0/614377323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 msgr2=0x563308f93d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe704097ce0 tx=0x7fe704010e30 comp rx=0 tx=0).stop 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- 172.21.15.123:0/614377323 shutdown_connections 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9f3a0 0x563308fa1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9e9e0 0x563308faf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 --2- 172.21.15.123:0/614377323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308f93d40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- 172.21.15.123:0/614377323 >> 172.21.15.123:0/614377323 conn(0x563308f98c00 msgr2=0x563308f9c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- 172.21.15.123:0/614377323 shutdown_connections 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- 172.21.15.123:0/614377323 wait complete. 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 -- start start 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.076+0000 7fe71da1d740 1 Processor -- start 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9e9e0 0x563308faf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9f3a0 0x563308fab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141a99b0 con 0x563308f9f3a0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141aacd0 con 0x563308f9d810 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141aa010 con 0x563308f9e9e0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe71da1d740 10 client.0 _ll_register_callbacks cb 0x7fe70c044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9e9e0 0x563308faf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:33856/0 (socket says 172.21.15.123:33856) 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 -- 172.21.15.123:0/728279734 learned_addr learned my addr 172.21.15.123:0/728279734 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe718e00640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9f3a0 0x563308fab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 -- 172.21.15.123:0/728279734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9e9e0 msgr2=0x563308faf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9e9e0 0x563308faf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 -- 172.21.15.123:0/728279734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9f3a0 msgr2=0x563308fab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9f3a0 0x563308fab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.077+0000 7fe713e00640 1 -- 172.21.15.123:0/728279734 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563308fabe50 con 0x563308f9d810 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe713e00640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe7040cc4c0 tx=0x7fe7040d6cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=381ef1e68ebb9cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:05.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7040dd040 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe704099610 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563308f0f800 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe70409b0e0 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 10 client.26933 fetch_fsmap learned FSMap version 19 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56330900fc10 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563308f0f800 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe70407e070 con 0x563308f9d810 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 10 client.26933 fetch_fsmap finished waiting for FSMap version 19 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 20 client.26933 populate_metadata read hostname 'smithi123' 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563309010000 0x563309012420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 4 client.26933 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.078+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 --> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563308f09b50 con 0x563309010000 2024-03-24T00:50:05.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe713400640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563309010000 0x563309012420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe713400640 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563309010000 0x563309012420 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183484015ebb476a server_cookie=a929cdc5ca64464c in_seq=0 out_seq=0 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe6fe000640 10 client.26933 ms_handle_connect on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe6fe000640 1 -- 172.21.15.123:0/728279734 <== mds.0 v2:172.21.15.177:6838/2781882004 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563308f09b50 con 0x563309010000 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe6fe000640 10 client.26933 handle_command_reply: tid=0 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe71da1d740 1 client.26933 shutdown 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 msgr2=0x563308faeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.079+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe7040cc4c0 tx=0x7fe7040d6cc0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 shutdown_connections 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563308f9f3a0 0x563308fab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:6838/2781882004,v1:172.21.15.177:6839/2781882004] conn(0x563309010000 0x563309012420 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563308f9e9e0 0x563308faf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 --2- 172.21.15.123:0/728279734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563308f9d810 0x563308faeb00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 >> 172.21.15.123:0/728279734 conn(0x563308f98c00 msgr2=0x563308f97620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe6fe000640 0 client.26933 ms_handle_reset on v2:172.21.15.177:6838/2781882004 2024-03-24T00:50:05.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 shutdown_connections 2024-03-24T00:50:05.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 1 -- 172.21.15.123:0/728279734 wait complete. 2024-03-24T00:50:05.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71da1d740 20 client.26933 trim_cache size 0 max 16384 2024-03-24T00:50:05.086 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 msgr2=0x7fe6f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 0x7fe6f80a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe70c02c630 tx=0x7fe70c032590 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 msgr2=0x7fe714199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.080+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe70400d950 tx=0x7fe70400de10 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 shutdown_connections 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe714105490 0x7fe7141a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe6f80a2800 0x7fe6f80a4cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe714104ad0 0x7fe71419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 --2- 172.21.15.123:0/54959467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7141038e0 0x7fe714199b70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 >> 172.21.15.123:0/54959467 conn(0x7fe7140fee50 msgr2=0x7fe714107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 shutdown_connections 2024-03-24T00:50:05.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:05.081+0000 7fe71ae00640 1 -- 172.21.15.123:0/54959467 wait complete. 2024-03-24T00:50:05.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:05 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3137664629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:05.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:05 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3137664629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:05.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:05 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3137664629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:06.099 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-24T00:50:06.100 INFO:tasks.fwd_scrub:done joining 2024-03-24T00:50:06.100 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T00:50:06.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:50:06.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 -- 172.21.15.123:0/4079879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1071d0 msgr2=0x7f4eac109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 --2- 172.21.15.123:0/4079879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1071d0 0x7f4eac109620 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009ca0 tx=0x7f4ea402c100 comp rx=0 tx=0).stop 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 -- 172.21.15.123:0/4079879978 shutdown_connections 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 --2- 172.21.15.123:0/4079879978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1071d0 0x7f4eac109620 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 --2- 172.21.15.123:0/4079879978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1048a0 0x7f4eac106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 --2- 172.21.15.123:0/4079879978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 -- 172.21.15.123:0/4079879978 >> 172.21.15.123:0/4079879978 conn(0x7f4eac0fb980 msgr2=0x7f4eac0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 -- 172.21.15.123:0/4079879978 shutdown_connections 2024-03-24T00:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.329+0000 7f4eb1400640 1 -- 172.21.15.123:0/4079879978 wait complete. 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.330+0000 7f4eb1400640 1 Processor -- start 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.330+0000 7f4eb1400640 1 -- start start 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1048a0 0x7f4eac0fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1071d0 0x7f4eac100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac10d180 con 0x7f4eac1071d0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac10d000 con 0x7f4eac1048a0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb1400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac10d300 con 0x7f4eac101f70 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1048a0 0x7f4eac0fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:38924/0 (socket says 172.21.15.123:38924) 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 -- 172.21.15.123:0/9701949 learned_addr learned my addr 172.21.15.123:0/9701949 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eb0a00640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1071d0 0x7f4eac100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 -- 172.21.15.123:0/9701949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1048a0 msgr2=0x7f4eac0fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1048a0 0x7f4eac0fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 -- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1071d0 msgr2=0x7f4eac100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1071d0 0x7f4eac100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.331+0000 7f4eabe00640 1 -- 172.21.15.123:0/9701949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac100c80 con 0x7f4eac101f70 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4eabe00640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9f0 tx=0x7f4e9c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00cd80 con 0x7f4eac101f70 2024-03-24T00:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e9c005500 con 0x7f4eac101f70 2024-03-24T00:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac1a18b0 con 0x7f4eac101f70 2024-03-24T00:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c010690 con 0x7f4eac101f70 2024-03-24T00:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.332+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac1a1db0 con 0x7f4eac101f70 2024-03-24T00:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.333+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac0fd470 con 0x7f4eac101f70 2024-03-24T00:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.335+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4e9c0046c0 con 0x7f4eac101f70 2024-03-24T00:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.336+0000 7f4ea8c00640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 0x7f4e900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.336+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4e9c0ceea0 con 0x7f4eac101f70 2024-03-24T00:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.337+0000 7f4eab400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 0x7f4e900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.337+0000 7f4eab400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 0x7f4e900a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4eac1015e0 tx=0x7f4e98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:06.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.340+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e9c09bef0 con 0x7f4eac101f70 2024-03-24T00:50:06.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.545+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4eac0622a0 con 0x7f4eac101f70 2024-03-24T00:50:06.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.547+0000 7f4ea8c00640 1 -- 172.21.15.123:0/9701949 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f4e9c09b7f0 con 0x7f4eac101f70 2024-03-24T00:50:06.548 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:06.549 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 msgr2=0x7f4e900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 0x7f4e900a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4eac1015e0 tx=0x7f4e98005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 msgr2=0x7f4eac101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9f0 tx=0x7f4e9c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 shutdown_connections 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f4e900a2800 0x7f4e900a4cc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eac1071d0 0x7f4eac100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eac1048a0 0x7f4eac0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 --2- 172.21.15.123:0/9701949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4eac101f70 0x7f4eac101870 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 >> 172.21.15.123:0/9701949 conn(0x7f4eac0fb980 msgr2=0x7f4eac102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:06.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 shutdown_connections 2024-03-24T00:50:06.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.550+0000 7f4eb1400640 1 -- 172.21.15.123:0/9701949 wait complete. 2024-03-24T00:50:06.571 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:50:06.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.770+0000 7f986fe00640 1 -- 172.21.15.123:0/316499462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870102000 msgr2=0x7f98701043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.770+0000 7f986fe00640 1 --2- 172.21.15.123:0/316499462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870102000 0x7f98701043f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9860009900 tx=0x7f986002bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 -- 172.21.15.123:0/316499462 shutdown_connections 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 --2- 172.21.15.123:0/316499462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870107260 0x7f98701096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 --2- 172.21.15.123:0/316499462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f9870106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 --2- 172.21.15.123:0/316499462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870102000 0x7f98701043f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 -- 172.21.15.123:0/316499462 >> 172.21.15.123:0/316499462 conn(0x7f98700fba10 msgr2=0x7f98700fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 -- 172.21.15.123:0/316499462 shutdown_connections 2024-03-24T00:50:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.771+0000 7f986fe00640 1 -- 172.21.15.123:0/316499462 wait complete. 2024-03-24T00:50:06.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.772+0000 7f986fe00640 1 Processor -- start 2024-03-24T00:50:06.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.772+0000 7f986fe00640 1 -- start start 2024-03-24T00:50:06.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.772+0000 7f986fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 0x7f9870101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f98700fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 0x7f98701004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987010d260 con 0x7f9870102000 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987010d0e0 con 0x7f9870104930 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987010d3e0 con 0x7f9870107260 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 0x7f9870101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f98700fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 0x7f98701004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 0x7f9870101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46342/0 (socket says 172.21.15.123:46342) 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986ea00640 1 -- 172.21.15.123:0/961759209 learned_addr learned my addr 172.21.15.123:0/961759209 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 -- 172.21.15.123:0/961759209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 msgr2=0x7f98701004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 0x7f98701004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 -- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 msgr2=0x7f9870101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 0x7f9870101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986e000640 1 -- 172.21.15.123:0/961759209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870100a30 con 0x7f9870104930 2024-03-24T00:50:06.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.773+0000 7f986f400640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 0x7f98701004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:06.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f986e000640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f98700fffb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f985c00e9c0 tx=0x7f985c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:06.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c00ccc0 con 0x7f9870104930 2024-03-24T00:50:06.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f985c010910 con 0x7f9870104930 2024-03-24T00:50:06.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9870100d20 con 0x7f9870104930 2024-03-24T00:50:06.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c0188c0 con 0x7f9870104930 2024-03-24T00:50:06.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.774+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98701a63c0 con 0x7f9870104930 2024-03-24T00:50:06.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.775+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98700fd500 con 0x7f9870104930 2024-03-24T00:50:06.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.778+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f985c010ac0 con 0x7f9870104930 2024-03-24T00:50:06.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.779+0000 7f9867400640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 0x7f98540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:06.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.779+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f985c0cede0 con 0x7f9870104930 2024-03-24T00:50:06.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.779+0000 7f986ea00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 0x7f98540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:06.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.780+0000 7f986ea00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 0x7f98540a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f986002c8e0 tx=0x7f986002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:06.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.783+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f985c09b840 con 0x7f9870104930 2024-03-24T00:50:06.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:06 smithi204 ceph-mon[36345]: pgmap v751: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-24T00:50:06.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:06 smithi123 ceph-mon[31487]: pgmap v751: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-24T00:50:06.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:06 smithi177 ceph-mon[36034]: pgmap v751: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-24T00:50:06.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.983+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98700622a0 con 0x7f9870104930 2024-03-24T00:50:06.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.983+0000 7f9867400640 1 -- 172.21.15.123:0/961759209 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f985c09f8e0 con 0x7f9870104930 2024-03-24T00:50:06.984 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:50:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 msgr2=0x7f98540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 0x7f98540a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f986002c8e0 tx=0x7f986002df90 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 msgr2=0x7f98700fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f98700fffb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f985c00e9c0 tx=0x7f985c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 shutdown_connections 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9870107260 0x7f98701004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870104930 0x7f98700fffb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f98540a2800 0x7f98540a4cc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 --2- 172.21.15.123:0/961759209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870102000 0x7f9870101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 >> 172.21.15.123:0/961759209 conn(0x7f98700fba10 msgr2=0x7f9870102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 shutdown_connections 2024-03-24T00:50:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:06.987+0000 7f986fe00640 1 -- 172.21.15.123:0/961759209 wait complete. 2024-03-24T00:50:07.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:50:07.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 -- 172.21.15.123:0/3377320461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed00103260 msgr2=0x7fed00109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 --2- 172.21.15.123:0/3377320461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed00103260 0x7fed00109af0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009ca0 tx=0x7fecfc02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 -- 172.21.15.123:0/3377320461 shutdown_connections 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 --2- 172.21.15.123:0/3377320461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed00103260 0x7fed00109af0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 --2- 172.21.15.123:0/3377320461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed001028a0 0x7fed00102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 --2- 172.21.15.123:0/3377320461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 0x7fed00101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 -- 172.21.15.123:0/3377320461 >> 172.21.15.123:0/3377320461 conn(0x7fed000fcc40 msgr2=0x7fed000ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 -- 172.21.15.123:0/3377320461 shutdown_connections 2024-03-24T00:50:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.203+0000 7fed08800640 1 -- 172.21.15.123:0/3377320461 wait complete. 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.204+0000 7fed08800640 1 Processor -- start 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.204+0000 7fed08800640 1 -- start start 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.204+0000 7fed08800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 0x7fed00199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.204+0000 7fed08800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed08800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00103260 0x7fed001a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed08800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0010d310 con 0x7fed00103260 2024-03-24T00:50:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed08800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0010d190 con 0x7fed001028a0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed08800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0010d490 con 0x7fed001016b0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 0x7fed00199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed06800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00103260 0x7fed001a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51028/0 (socket says 172.21.15.123:51028) 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 -- 172.21.15.123:0/1674913791 learned_addr learned my addr 172.21.15.123:0/1674913791 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 -- 172.21.15.123:0/1674913791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 msgr2=0x7fed00199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 0x7fed00199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 -- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00103260 msgr2=0x7fed001a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00103260 0x7fed001a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 -- 172.21.15.123:0/1674913791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed001a17c0 con 0x7fed001028a0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.205+0000 7fed05400640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e9f0 tx=0x7fecf000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.206+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf000cd80 con 0x7fed001028a0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.206+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecf0010910 con 0x7fed001028a0 2024-03-24T00:50:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.206+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0018990 con 0x7fed001028a0 2024-03-24T00:50:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.206+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed001a1ab0 con 0x7fed001028a0 2024-03-24T00:50:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.206+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed001a2050 con 0x7fed001028a0 2024-03-24T00:50:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.207+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed001a2220 con 0x7fed001028a0 2024-03-24T00:50:07.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.210+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fecf0010ac0 con 0x7fed001028a0 2024-03-24T00:50:07.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.211+0000 7fecfaa00640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 0x7fece80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.211+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fecf00cee50 con 0x7fed001028a0 2024-03-24T00:50:07.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.211+0000 7fed05e00640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 0x7fece80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.212+0000 7fed05e00640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 0x7fece80a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fed00102700 tx=0x7fecf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:07.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.217+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fecf009b8b0 con 0x7fed001028a0 2024-03-24T00:50:07.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.419+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed001a2490 con 0x7fed001028a0 2024-03-24T00:50:07.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.420+0000 7fecfaa00640 1 -- 172.21.15.123:0/1674913791 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fecf00cce20 con 0x7fed001028a0 2024-03-24T00:50:07.421 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 msgr2=0x7fece80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 0x7fece80a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fed00102700 tx=0x7fecf4005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 msgr2=0x7fed0019a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e9f0 tx=0x7fecf000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 shutdown_connections 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fece80a2880 0x7fece80a4d40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00103260 0x7fed001a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed001028a0 0x7fed0019a0e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 --2- 172.21.15.123:0/1674913791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed001016b0 0x7fed00199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 >> 172.21.15.123:0/1674913791 conn(0x7fed000fcc40 msgr2=0x7fed000738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:07.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 shutdown_connections 2024-03-24T00:50:07.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.423+0000 7fed08800640 1 -- 172.21.15.123:0/1674913791 wait complete. 2024-03-24T00:50:07.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 -- 172.21.15.123:0/158472509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c102130 msgr2=0x7ff39c104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 --2- 172.21.15.123:0/158472509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c102130 0x7ff39c104520 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009900 tx=0x7ff38c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 -- 172.21.15.123:0/158472509 shutdown_connections 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 --2- 172.21.15.123:0/158472509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c107390 0x7ff39c1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 --2- 172.21.15.123:0/158472509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 --2- 172.21.15.123:0/158472509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c102130 0x7ff39c104520 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.622+0000 7ff3a3200640 1 -- 172.21.15.123:0/158472509 >> 172.21.15.123:0/158472509 conn(0x7ff39c0fbb40 msgr2=0x7ff39c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.623+0000 7ff3a3200640 1 -- 172.21.15.123:0/158472509 shutdown_connections 2024-03-24T00:50:07.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.623+0000 7ff3a3200640 1 -- 172.21.15.123:0/158472509 wait complete. 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.623+0000 7ff3a3200640 1 Processor -- start 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 -- start start 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 0x7ff39c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c10d3b0 con 0x7ff39c102130 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c10d230 con 0x7ff39c107390 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c10d530 con 0x7ff39c104a60 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 0x7ff39c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:51048/0 (socket says 172.21.15.123:51048) 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:38982/0 (socket says 172.21.15.123:38982) 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a1200640 1 -- 172.21.15.123:0/3529444769 learned_addr learned my addr 172.21.15.123:0/3529444769 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 0x7ff39c199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46388/0 (socket says 172.21.15.123:46388) 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 -- 172.21.15.123:0/3529444769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 msgr2=0x7ff39c1a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 -- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 msgr2=0x7ff39c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 0x7ff39c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff39b400640 1 -- 172.21.15.123:0/3529444769 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c1a1890 con 0x7ff39c104a60 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.624+0000 7ff3a1200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff39b400640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9c0 tx=0x7ff38800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38800ccc0 con 0x7ff39c104a60 2024-03-24T00:50:07.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff388005020 con 0x7ff39c104a60 2024-03-24T00:50:07.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388004a00 con 0x7ff39c104a60 2024-03-24T00:50:07.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c1a1b80 con 0x7ff39c104a60 2024-03-24T00:50:07.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.625+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c1a2050 con 0x7ff39c104a60 2024-03-24T00:50:07.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.626+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c108ee0 con 0x7ff39c104a60 2024-03-24T00:50:07.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.628+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3880051d0 con 0x7ff39c104a60 2024-03-24T00:50:07.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.630+0000 7ff398c00640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 0x7ff3800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:07.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.630+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff3880ce9e0 con 0x7ff39c104a60 2024-03-24T00:50:07.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.630+0000 7ff39be00640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 0x7ff3800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:07.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.631+0000 7ff39be00640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 0x7ff3800a4d40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff38c02c240 tx=0x7ff38c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:07.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.634+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff38809ba30 con 0x7ff39c104a60 2024-03-24T00:50:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:07 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/9701949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:07 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/961759209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:07.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:07 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1674913791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:07.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.832+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff39c02d070 con 0x7ff39c104a60 2024-03-24T00:50:07.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.833+0000 7ff398c00640 1 -- 172.21.15.123:0/3529444769 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff38809b330 con 0x7ff39c104a60 2024-03-24T00:50:07.833 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.835+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 msgr2=0x7ff3800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.835+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 0x7ff3800a4d40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff38c02c240 tx=0x7ff38c02df90 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.835+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 msgr2=0x7ff39c19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.835+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9c0 tx=0x7ff38800ee80 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 shutdown_connections 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff39c107390 0x7ff39c1a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff39c104a60 0x7ff39c19a1c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7ff3800a2880 0x7ff3800a4d40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 --2- 172.21.15.123:0/3529444769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff39c102130 0x7ff39c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 >> 172.21.15.123:0/3529444769 conn(0x7ff39c0fbb40 msgr2=0x7ff39c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 shutdown_connections 2024-03-24T00:50:07.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:07.836+0000 7ff3a3200640 1 -- 172.21.15.123:0/3529444769 wait complete. 2024-03-24T00:50:07.851 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:07 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/9701949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:07.851 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:07 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/961759209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:07.851 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:07 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1674913791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:07.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T00:50:07.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:07 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/9701949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:07.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:07 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/961759209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:07.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:07 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1674913791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.038+0000 7fca31200640 1 -- 172.21.15.123:0/3493330369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1048e0 msgr2=0x7fca2c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.038+0000 7fca31200640 1 --2- 172.21.15.123:0/3493330369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1048e0 0x7fca2c104d60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca18009900 tx=0x7fca1802bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 -- 172.21.15.123:0/3493330369 shutdown_connections 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 --2- 172.21.15.123:0/3493330369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1052a0 0x7fca2c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 --2- 172.21.15.123:0/3493330369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1048e0 0x7fca2c104d60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 --2- 172.21.15.123:0/3493330369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 -- 172.21.15.123:0/3493330369 >> 172.21.15.123:0/3493330369 conn(0x7fca2c0fec80 msgr2=0x7fca2c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 -- 172.21.15.123:0/3493330369 shutdown_connections 2024-03-24T00:50:08.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.039+0000 7fca31200640 1 -- 172.21.15.123:0/3493330369 wait complete. 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 Processor -- start 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 -- start start 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 0x7fca2c199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1052a0 0x7fca2c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c10f3c0 con 0x7fca2c1036f0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c10f240 con 0x7fca2c1052a0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca31200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c10f540 con 0x7fca2c1048e0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca2b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca2aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 0x7fca2c199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1052a0 0x7fca2c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca2aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 0x7fca2c199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:39012/0 (socket says 172.21.15.123:39012) 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.040+0000 7fca2aa00640 1 -- 172.21.15.123:0/1578665376 learned_addr learned my addr 172.21.15.123:0/1578665376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 -- 172.21.15.123:0/1578665376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 msgr2=0x7fca2c199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 0x7fca2c199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 -- 172.21.15.123:0/1578665376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1052a0 msgr2=0x7fca2c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1052a0 0x7fca2c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 -- 172.21.15.123:0/1578665376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c1a15d0 con 0x7fca2c1036f0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca2b400640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c1999b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9c0 tx=0x7fca1c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00ccc0 con 0x7fca2c1036f0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c1a18c0 con 0x7fca2c1036f0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca1c0035f0 con 0x7fca2c1036f0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00ce60 con 0x7fca2c1036f0 2024-03-24T00:50:08.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.041+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c1a1de0 con 0x7fca2c1036f0 2024-03-24T00:50:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.042+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c10a9b0 con 0x7fca2c1036f0 2024-03-24T00:50:08.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.044+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fca1c0037a0 con 0x7fca2c1036f0 2024-03-24T00:50:08.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.046+0000 7fca23e00640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.046+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fca1c0ceed0 con 0x7fca2c1036f0 2024-03-24T00:50:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.046+0000 7fca2aa00640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.047+0000 7fca2aa00640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 0x7fca100a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fca1802c8e0 tx=0x7fca1802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.050+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca1c09bf20 con 0x7fca2c1036f0 2024-03-24T00:50:08.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.301+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fca2c02d070 con 0x7fca2c1036f0 2024-03-24T00:50:08.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.313+0000 7fca23e00640 1 -- 172.21.15.123:0/1578665376 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+739837 (secure 0 0 0) 0x7fca1c09b820 con 0x7fca2c1036f0 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "epoch": 18, 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "active_gid": 14152, 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "active_name": "x", 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "active_addrs": { 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.326 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6800", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2933502234 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6801", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2933502234 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "active_addr": "172.21.15.123:6801/2933502234", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "active_change": "2024-03-24T00:27:40.925387+0000", 2024-03-24T00:50:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "available": true, 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "standbys": [ 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 24112, 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "z", 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_features": 4540701547738038271, 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "available_modules": [ 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "alerts", 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.328 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.329 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T00:50:08.330 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.330 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.330 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.330 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.331 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.332 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.333 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.334 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.335 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.337 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_destination": { 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_destination", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email address to send alerts to", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.338 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_from_name": { 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_from_name", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "Ceph", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email From: name", 2024-03-24T00:50:08.339 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_host": { 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_host", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP server", 2024-03-24T00:50:08.340 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_password": { 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_password", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Password to authenticate with", 2024-03-24T00:50:08.341 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_port": { 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_port", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "465", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.342 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP port", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_sender": { 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_sender", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.343 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP envelope sender", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_ssl": { 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_ssl", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.344 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_user": { 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_user", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.345 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "User to authenticate as", 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.346 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "balancer", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "active": { 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "active", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.347 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_time": { 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_time", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0000", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.348 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_weekday": { 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_weekday", 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.349 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:08.350 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_max_iterations": { 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_max_iterations", 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "25", 2024-03-24T00:50:08.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "250", 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_metrics": { 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_metrics", 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.353 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_step": { 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_step", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.5", 2024-03-24T00:50:08.354 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0.001", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "0.999", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "aggressiveness of optimization", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "end_time": { 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_time", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.355 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2359", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "end_weekday": { 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_weekday", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.356 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.357 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.358 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.360 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.361 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.362 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "min_score": { 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_score", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.363 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "upmap", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.364 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-compat", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "none", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "read", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap-read" 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Balancer mode", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_ids": { 2024-03-24T00:50:08.365 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_ids", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.366 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_deviation": { 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_deviation", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.368 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_optimizations": { 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_optimizations", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.369 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm", 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_down_multiplier": { 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_down_multiplier", 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.370 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_refresh_rate": { 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_refresh_rate", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.371 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "20", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_starting_port": { 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_starting_port", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.372 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4721", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ptrace": { 2024-03-24T00:50:08.373 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_ptrace", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_interval": { 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_interval", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.375 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_memory_target_ratio": { 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.7", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.376 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_log_destination": { 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm_log_destination", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "file", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "file,syslog", 2024-03-24T00:50:08.377 INFO:teuthology.orchestra.run.smithi123.stdout: "syslog" 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "cgroups_split": { 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cgroups_split", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "config_checks_enabled": { 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_checks_enabled", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.379 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "config_dashboard": { 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_dashboard", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.380 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_alertmanager": { 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_alertmanager", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.381 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_base": { 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_base", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.382 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Container image name, without the tag", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_elasticsearch": { 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_elasticsearch", 2024-03-24T00:50:08.383 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "elasticsearch container image", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_grafana": { 2024-03-24T00:50:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_grafana", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_haproxy": { 2024-03-24T00:50:08.385 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_haproxy", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "HAproxy container image", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_agent": { 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_agent", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger agent container image", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.387 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_collector": { 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_collector", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger collector container image", 2024-03-24T00:50:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_query": { 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_query", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger query container image", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_keepalived": { 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_keepalived", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.390 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Keepalived container image", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_loki": { 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_loki", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.391 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Loki container image", 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_node_exporter": { 2024-03-24T00:50:08.392 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_node_exporter", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.393 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nvmeof": { 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nvmeof", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.394 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nvme-of container image", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_prometheus": { 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_prometheus", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T00:50:08.395 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_promtail": { 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_promtail", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T00:50:08.396 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Promtail container image", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_snmp_gateway": { 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_snmp_gateway", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.397 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SNMP Gateway container image", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "container_init": { 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_init", 2024-03-24T00:50:08.398 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "daemon_cache_timeout": { 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "daemon_cache_timeout", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.400 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "default_cephadm_command_timeout": { 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "900", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.401 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "default_registry": { 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_registry", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.402 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "device_cache_timeout": { 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_cache_timeout", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.403 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1800", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache device inventory", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "device_enhanced_scan": { 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_enhanced_scan", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.404 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "facts_cache_timeout": { 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "facts_cache_timeout", 2024-03-24T00:50:08.405 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache host facts data", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "host_check_interval": { 2024-03-24T00:50:08.406 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "host_check_interval", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to perform a host check", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "hw_monitoring": { 2024-03-24T00:50:08.407 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hw_monitoring", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.408 INFO:teuthology.orchestra.run.smithi123.stdout: "inventory_list_all": { 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "inventory_list_all", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.410 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "log_refresh_metadata": { 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_refresh_metadata", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.411 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.412 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.413 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.414 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.415 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.416 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.417 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "*", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "max_count_per_host": { 2024-03-24T00:50:08.418 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_count_per_host", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of daemons per service per host", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd_draining_count": { 2024-03-24T00:50:08.419 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_osd_draining_count", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "migration_current": { 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "migration_current", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "internal - do not modify", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.421 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "root", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm-package", 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: "root" 2024-03-24T00:50:08.422 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "oob_default_addr": { 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "oob_default_addr", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "169.254.1.1", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.423 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "prometheus_alerts_path": { 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus_alerts_path", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T00:50:08.424 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_insecure": { 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_insecure", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_password": { 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_password", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.426 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_url": { 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_url", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_username": { 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_username", 2024-03-24T00:50:08.428 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "secure_monitoring_stack": { 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secure_monitoring_stack", 2024-03-24T00:50:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "service_discovery_port": { 2024-03-24T00:50:08.430 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "service_discovery_port", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8765", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "cephadm service discovery port", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_config_file": { 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_config_file", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.432 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "use_agent": { 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_agent", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.433 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "use_repo_digest": { 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_repo_digest", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.434 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_failed_host_check": { 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_failed_host_check", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.435 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_daemons": { 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_daemons", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.436 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_hosts": { 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_hosts", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.437 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crash", 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.438 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.439 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.440 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.441 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.442 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.443 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "retain_interval": { 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retain_interval", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "31536000", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.444 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_recent_interval": { 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_recent_interval", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1209600", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.445 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dashboard", 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.446 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.447 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_ENABLED": { 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.450 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.451 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.452 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.453 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.454 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.455 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.456 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.457 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T00:50:08.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T00:50:08.459 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T00:50:08.460 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.461 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_URL": { 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_URL", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.466 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.467 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.468 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.469 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.470 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "{}", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.471 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T00:50:08.472 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T00:50:08.473 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.475 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.476 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.477 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.478 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.479 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.480 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.481 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.482 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T00:50:08.483 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.485 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "45", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.486 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.487 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.488 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.489 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.490 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.491 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.492 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.493 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T00:50:08.494 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "cross_origin_url": { 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cross_origin_url", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.498 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "crt_file": { 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crt_file", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.499 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "debug": { 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "debug", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.500 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable/disable debug options", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "jwt_token_ttl": { 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "jwt_token_ttl", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "28800", 2024-03-24T00:50:08.501 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.502 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.503 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.504 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.505 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.506 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.507 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "motd": { 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "motd", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "The message of the day", 2024-03-24T00:50:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect_resolve_ip_addr": { 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.509 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.510 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8080", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl_server_port": { 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl_server_port", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8443", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.513 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "redirect", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect" 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.514 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.515 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "url_prefix": { 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url_prefix", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.516 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_monitoring": { 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_monitoring", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.517 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "monitor device health metrics", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.518 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.520 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.521 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.522 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "mark_out_threshold": { 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mark_out_threshold", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2419200", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": { 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_name", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "device_health_metrics", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.524 INFO:teuthology.orchestra.run.smithi123.stdout: "retention_period": { 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retention_period", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15552000", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain device health metrics", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_frequency": { 2024-03-24T00:50:08.525 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_frequency", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.526 INFO:teuthology.orchestra.run.smithi123.stdout: "self_heal": { 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "self_heal", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.527 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.528 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_threshold": { 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_threshold", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7257600", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.529 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "diskprediction_local", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.530 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.532 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.533 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.534 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "predict_interval": { 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predict_interval", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.535 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "predictor_model": { 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predictor_model", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "prophetstor", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.536 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "influx", 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": false, 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "influxdb python module not found", 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "batch_size": { 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "batch_size", 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.538 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5000", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "database": { 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "database", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "ceph", 2024-03-24T00:50:08.539 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": { 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hostname", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.540 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server hostname", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "30", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "5", 2024-03-24T00:50:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.542 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.543 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.545 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.546 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "password": { 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "password", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "password of InfluxDB server user", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.547 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "port": { 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "port", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8086", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server port", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "false", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.549 INFO:teuthology.orchestra.run.smithi123.stdout: "threads": { 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threads", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "32", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "username": { 2024-03-24T00:50:08.550 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "username", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "username of InfluxDB server user", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "verify_ssl": { 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "verify_ssl", 2024-03-24T00:50:08.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "true", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "insights", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.553 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.554 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.555 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.556 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "iostat", 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.558 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.559 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.561 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.562 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "k8sevents", 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_event_retention_days": { 2024-03-24T00:50:08.563 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ceph_event_retention_days", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "config_check_secs": { 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_check_secs", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.564 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "10", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.565 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.566 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.567 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.568 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.569 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "localpool", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.570 INFO:teuthology.orchestra.run.smithi123.stdout: "failure_domain": { 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "failure_domain", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "host", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "failure domain for any created local pool", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.571 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.574 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.575 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": { 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_size", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default min_size for any created local pool", 2024-03-24T00:50:08.576 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "num_rep": { 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "num_rep", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default replica count for any created local pool", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": { 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_num", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "128", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.578 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "prefix": { 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prefix", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name prefix for any created local pool", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.579 INFO:teuthology.orchestra.run.smithi123.stdout: "subtree": { 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subtree", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "rack", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mds_autoscaler", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.581 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.582 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.584 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.585 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirroring", 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.586 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.587 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.588 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.589 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "nfs", 2024-03-24T00:50:08.591 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.592 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.593 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.594 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.596 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "fail_fs": { 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fail_fs", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.597 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.598 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.599 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator": { 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "rook", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "test_orchestrator" 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Orchestrator backend", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.603 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_perf_query", 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.605 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.606 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.607 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.608 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_support", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.609 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.610 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.611 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.612 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.613 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_autoscaler", 2024-03-24T00:50:08.614 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.615 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.616 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.617 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.619 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "threshold": { 2024-03-24T00:50:08.620 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threshold", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1.0", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "scaling threshold", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "progress", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_pg_recovery_event": { 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_pg_recovery_event", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.622 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.623 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completed_events": { 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_completed_events", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.628 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "50", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "number of past completed events to remember", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long the module is going to sleep", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus", 2024-03-24T00:50:08.630 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "cache": { 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cache", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.631 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "exclude_perf_counters": { 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "exclude_perf_counters", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.632 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.633 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.634 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.635 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.636 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools": { 2024-03-24T00:50:08.637 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T00:50:08.638 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "300", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_interval": { 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_interval", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.639 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15.0", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.640 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.641 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "9283", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "stale_cache_strategy": { 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stale_cache_strategy", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.642 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "log", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.643 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "default", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "default", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "error" 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:08.644 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.646 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.647 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.648 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.650 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_snap_create": { 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_snap_create", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "mirror_snapshot_schedule": { 2024-03-24T00:50:08.651 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "trash_purge_schedule": { 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "trash_purge_schedule", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.652 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.653 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "restful", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_auth": { 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_auth", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.654 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.655 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.656 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.657 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.658 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.661 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.662 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rgw", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.663 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.664 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.665 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.666 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.667 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rook", 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.668 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.669 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.670 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.671 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.672 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "storage_class": { 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "storage_class", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "local", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "selftest", 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.674 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.675 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.676 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.677 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.678 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "roption1": { 2024-03-24T00:50:08.679 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption1", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "roption2": { 2024-03-24T00:50:08.680 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption2", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "xyz", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption1": { 2024-03-24T00:50:08.681 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption1", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption2": { 2024-03-24T00:50:08.682 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption2", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption3": { 2024-03-24T00:50:08.683 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption3", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption4": { 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption4", 2024-03-24T00:50:08.684 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption5": { 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption5", 2024-03-24T00:50:08.685 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption6": { 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption6", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.686 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption7": { 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption7", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.687 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "42", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "testkey": { 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testkey", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.688 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "testlkey": { 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testlkey", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.689 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "testnewline": { 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testnewline", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.690 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snap_schedule", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.691 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_m_granularity": { 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_m_granularity", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.692 INFO:teuthology.orchestra.run.smithi123.stdout: "dump_on_update": { 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dump_on_update", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "dump database to debug log on update", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.693 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.694 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.695 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.696 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.697 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stats", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.699 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.700 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.700 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.700 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.700 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.703 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.704 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.705 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "status", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.706 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.707 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.708 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.709 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.710 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telegraf", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "address": { 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "address", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.711 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.712 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.713 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.714 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.715 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.716 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.717 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.718 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telemetry", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_basic": { 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_basic", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.719 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_crash": { 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_crash", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.720 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_device": { 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_device", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.721 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_ident": { 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_ident", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.722 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_perf": { 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_perf", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.723 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "contact": { 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "contact", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.724 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "description": { 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "description", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "device_url": { 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_url", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.726 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.727 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "24", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "8", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.728 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "last_opt_revision": { 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "last_opt_revision", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.729 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard": { 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.730 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard_description": { 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard_description", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.731 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.732 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.733 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.734 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.735 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.736 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "organization": { 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "organization", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.737 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "proxy": { 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "proxy", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.738 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "url": { 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "test_orchestrator", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.740 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.741 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.742 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.743 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.744 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.745 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.746 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.748 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.749 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_clones": { 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_clones", 2024-03-24T00:50:08.750 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.751 INFO:teuthology.orchestra.run.smithi123.stdout: "periodic_async_work": { 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "periodic_async_work", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Periodically check for async work", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.752 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_delay": { 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_delay", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.753 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_no_wait": { 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.754 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "discovery_interval": { 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "discovery_interval", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "100", 2024-03-24T00:50:08.755 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "identifier": { 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "identifier", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.756 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.757 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.758 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.759 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.760 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.761 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.762 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_host": { 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_host", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.763 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_port": { 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_port", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10051", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.764 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_sender": { 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_sender", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.765 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 24116, 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "y", 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_features": 4540701547738038271, 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "available_modules": [ 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "alerts", 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.766 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.768 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.770 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.771 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_destination": { 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_destination", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.772 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email address to send alerts to", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_from_name": { 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_from_name", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "Ceph", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.773 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email From: name", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_host": { 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_host", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP server", 2024-03-24T00:50:08.774 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_password": { 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_password", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Password to authenticate with", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.775 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_port": { 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_port", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "465", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP port", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_sender": { 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_sender", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP envelope sender", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.777 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_ssl": { 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_ssl", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.778 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_user": { 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_user", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "User to authenticate as", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "balancer", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "active": { 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "active", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.780 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_time": { 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_time", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0000", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.781 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_weekday": { 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_weekday", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T00:50:08.782 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_max_iterations": { 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_max_iterations", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "25", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "250", 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.783 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_metrics": { 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_metrics", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.784 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_step": { 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_step", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.5", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0.001", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "0.999", 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.785 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "aggressiveness of optimization", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "end_time": { 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_time", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2359", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:08.786 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "end_weekday": { 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_weekday", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T00:50:08.787 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.788 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.789 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.790 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.791 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.792 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "min_score": { 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_score", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.793 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "upmap", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-compat", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "none", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "read", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap-read" 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Balancer mode", 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_ids": { 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_ids", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T00:50:08.796 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_deviation": { 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_deviation", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_optimizations": { 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_optimizations", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_down_multiplier": { 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_down_multiplier", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_refresh_rate": { 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_refresh_rate", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "20", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_starting_port": { 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_starting_port", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4721", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ptrace": { 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_ptrace", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_interval": { 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_interval", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_memory_target_ratio": { 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.7", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_log_destination": { 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm_log_destination", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi123.stdout: "file", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "file,syslog", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "syslog" 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "cgroups_split": { 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cgroups_split", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.808 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "config_checks_enabled": { 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_checks_enabled", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.809 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "config_dashboard": { 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_dashboard", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.810 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_alertmanager": { 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_alertmanager", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.811 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_base": { 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_base", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Container image name, without the tag", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.812 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_elasticsearch": { 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_elasticsearch", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "elasticsearch container image", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_grafana": { 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_grafana", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.814 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_haproxy": { 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_haproxy", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "HAproxy container image", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_agent": { 2024-03-24T00:50:08.815 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_agent", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger agent container image", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_collector": { 2024-03-24T00:50:08.816 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_collector", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger collector container image", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_query": { 2024-03-24T00:50:08.817 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_query", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger query container image", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_keepalived": { 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_keepalived", 2024-03-24T00:50:08.818 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Keepalived container image", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_loki": { 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_loki", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.819 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Loki container image", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_node_exporter": { 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_node_exporter", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.820 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nvmeof": { 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nvmeof", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.821 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nvme-of container image", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_prometheus": { 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_prometheus", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T00:50:08.822 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_promtail": { 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_promtail", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.823 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Promtail container image", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_snmp_gateway": { 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_snmp_gateway", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.824 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SNMP Gateway container image", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "container_init": { 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_init", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.825 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "daemon_cache_timeout": { 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "daemon_cache_timeout", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.826 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "default_cephadm_command_timeout": { 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "900", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.827 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "default_registry": { 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_registry", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.828 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "device_cache_timeout": { 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_cache_timeout", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1800", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache device inventory", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.829 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "device_enhanced_scan": { 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_enhanced_scan", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "facts_cache_timeout": { 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "facts_cache_timeout", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache host facts data", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.831 INFO:teuthology.orchestra.run.smithi123.stdout: "host_check_interval": { 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "host_check_interval", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to perform a host check", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "hw_monitoring": { 2024-03-24T00:50:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hw_monitoring", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "inventory_list_all": { 2024-03-24T00:50:08.833 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "inventory_list_all", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.834 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.835 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "log_refresh_metadata": { 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_refresh_metadata", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T00:50:08.836 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.837 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.838 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.839 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.840 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "*", 2024-03-24T00:50:08.841 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "max_count_per_host": { 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_count_per_host", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.842 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of daemons per service per host", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd_draining_count": { 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_osd_draining_count", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.843 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "migration_current": { 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "migration_current", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.844 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "internal - do not modify", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "root", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.845 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm-package", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "root" 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "oob_default_addr": { 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "oob_default_addr", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.846 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "169.254.1.1", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "prometheus_alerts_path": { 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus_alerts_path", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T00:50:08.847 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_insecure": { 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_insecure", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.848 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_password": { 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_password", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.849 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_url": { 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_url", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T00:50:08.850 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_username": { 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_username", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T00:50:08.851 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "secure_monitoring_stack": { 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secure_monitoring_stack", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.852 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "service_discovery_port": { 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "service_discovery_port", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8765", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "cephadm service discovery port", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_config_file": { 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_config_file", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.854 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "use_agent": { 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_agent", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "use_repo_digest": { 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_repo_digest", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.856 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_failed_host_check": { 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_failed_host_check", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_daemons": { 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_daemons", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.858 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_hosts": { 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_hosts", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crash", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.860 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.861 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.862 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.863 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.864 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "retain_interval": { 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retain_interval", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "31536000", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.865 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_recent_interval": { 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_recent_interval", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1209600", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.866 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dashboard", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.867 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.868 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.869 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_ENABLED": { 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.870 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.871 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.872 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.875 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.876 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T00:50:08.877 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T00:50:08.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.879 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.880 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.881 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_URL": { 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_URL", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.882 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.883 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.884 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.885 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.886 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.887 INFO:teuthology.orchestra.run.smithi123.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "{}", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T00:50:08.888 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.889 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.890 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.892 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.893 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.894 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.895 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.897 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.897 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.898 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.899 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T00:50:08.901 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.902 INFO:teuthology.orchestra.run.smithi123.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "45", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T00:50:08.903 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.904 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.906 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.907 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.908 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.909 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.910 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "cross_origin_url": { 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cross_origin_url", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.911 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "crt_file": { 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crt_file", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.912 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "debug": { 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "debug", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.913 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable/disable debug options", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "jwt_token_ttl": { 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "jwt_token_ttl", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "28800", 2024-03-24T00:50:08.914 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.915 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.917 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.919 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.920 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "motd": { 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "motd", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "The message of the day", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.921 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect_resolve_ip_addr": { 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.922 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.923 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8080", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.924 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.925 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl_server_port": { 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl_server_port", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8443", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "redirect", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect" 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.928 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "url_prefix": { 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url_prefix", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.929 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_monitoring": { 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_monitoring", 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.930 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "monitor device health metrics", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.931 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.932 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.933 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.935 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "mark_out_threshold": { 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mark_out_threshold", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2419200", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.936 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": { 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_name", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "device_health_metrics", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "retention_period": { 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retention_period", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15552000", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain device health metrics", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.938 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_frequency": { 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_frequency", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.939 INFO:teuthology.orchestra.run.smithi123.stdout: "self_heal": { 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "self_heal", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.940 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_threshold": { 2024-03-24T00:50:08.941 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_threshold", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7257600", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "diskprediction_local", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.943 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.944 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.946 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.947 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "predict_interval": { 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predict_interval", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.948 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "predictor_model": { 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predictor_model", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "prophetstor", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "influx", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": false, 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "influxdb python module not found", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "batch_size": { 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "batch_size", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5000", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.951 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "database": { 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "database", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "ceph", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.952 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": { 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hostname", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.953 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server hostname", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "30", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "5", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T00:50:08.954 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.956 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.957 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.959 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "password": { 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "password", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "password of InfluxDB server user", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "port": { 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "port", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8086", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server port", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:08.961 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "false", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "threads": { 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threads", 2024-03-24T00:50:08.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "32", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "username": { 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "username", 2024-03-24T00:50:08.963 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "username of InfluxDB server user", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "verify_ssl": { 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "verify_ssl", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.964 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "true", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "insights", 2024-03-24T00:50:08.965 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.966 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.967 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.968 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.969 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.970 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "iostat", 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.971 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.972 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.975 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "k8sevents", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_event_retention_days": { 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ceph_event_retention_days", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.977 INFO:teuthology.orchestra.run.smithi123.stdout: "config_check_secs": { 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_check_secs", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "10", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.978 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.979 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.980 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.981 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.983 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "localpool", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "failure_domain": { 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "failure_domain", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "host", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "failure domain for any created local pool", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.984 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": { 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_size", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default min_size for any created local pool", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "num_rep": { 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "num_rep", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default replica count for any created local pool", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": { 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_num", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "128", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.991 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "prefix": { 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prefix", 2024-03-24T00:50:08.992 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name prefix for any created local pool", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "subtree": { 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subtree", 2024-03-24T00:50:08.993 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "rack", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:08.994 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mds_autoscaler", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.995 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.996 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:08.999 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirroring", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.001 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.002 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.003 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.004 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.005 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "nfs", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.006 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.007 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.008 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.009 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.010 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.011 INFO:teuthology.orchestra.run.smithi123.stdout: "fail_fs": { 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fail_fs", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.012 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.013 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.014 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.015 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.016 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator": { 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.017 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "rook", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "test_orchestrator" 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Orchestrator backend", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_perf_query", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.019 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.020 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.021 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.022 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.023 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_support", 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.024 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.025 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.026 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.027 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.028 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_autoscaler", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.029 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.030 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.031 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.032 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.033 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.034 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.035 INFO:teuthology.orchestra.run.smithi123.stdout: "threshold": { 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threshold", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1.0", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "scaling threshold", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "progress", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_pg_recovery_event": { 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_pg_recovery_event", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.037 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.038 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.040 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.043 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completed_events": { 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_completed_events", 2024-03-24T00:50:09.044 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "50", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "number of past completed events to remember", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.045 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long the module is going to sleep", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus", 2024-03-24T00:50:09.046 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "cache": { 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cache", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.047 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "exclude_perf_counters": { 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "exclude_perf_counters", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.048 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.049 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.050 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.051 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.053 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools": { 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.054 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "300", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.055 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_interval": { 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_interval", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15.0", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.056 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.057 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "9283", 2024-03-24T00:50:09.058 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "stale_cache_strategy": { 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stale_cache_strategy", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.059 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "log", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.060 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "default", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "default", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "error" 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.061 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.062 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.064 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.065 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.066 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.067 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.068 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_snap_create": { 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_snap_create", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.069 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "mirror_snapshot_schedule": { 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.070 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "trash_purge_schedule": { 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "trash_purge_schedule", 2024-03-24T00:50:09.071 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.072 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "restful", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_auth": { 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_auth", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.073 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.074 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.075 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.076 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.077 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.078 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.079 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.080 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.081 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.082 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rgw", 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.083 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rook", 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "storage_class": { 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "storage_class", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "local", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "selftest", 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "roption1": { 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption1", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "roption2": { 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption2", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "xyz", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption1": { 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption1", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption2": { 2024-03-24T00:50:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption2", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption3": { 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption3", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption4": { 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption4", 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption5": { 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption5", 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption6": { 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption6", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption7": { 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption7", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "42", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "testkey": { 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testkey", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "testlkey": { 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testlkey", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "testnewline": { 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testnewline", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snap_schedule", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_m_granularity": { 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_m_granularity", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "dump_on_update": { 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dump_on_update", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "dump database to debug log on update", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stats", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "status", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telegraf", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "address": { 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "address", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telemetry", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_basic": { 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_basic", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_crash": { 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_crash", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T00:50:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_device": { 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_device", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_ident": { 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_ident", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_perf": { 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_perf", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "contact": { 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "contact", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "description": { 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "description", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "device_url": { 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_url", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "24", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "8", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "last_opt_revision": { 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "last_opt_revision", 2024-03-24T00:50:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard": { 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard", 2024-03-24T00:50:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard_description": { 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard_description", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "organization": { 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "organization", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "proxy": { 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "proxy", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "url": { 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "test_orchestrator", 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_clones": { 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_clones", 2024-03-24T00:50:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "periodic_async_work": { 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "periodic_async_work", 2024-03-24T00:50:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Periodically check for async work", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_delay": { 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_delay", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_no_wait": { 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix", 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "discovery_interval": { 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "discovery_interval", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "100", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "identifier": { 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "identifier", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_host": { 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_host", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_port": { 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_port", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10051", 2024-03-24T00:50:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_sender": { 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_sender", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "modules": [ 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "dashboard", 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "iostat", 2024-03-24T00:50:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "nfs", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "restful" 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "available_modules": [ 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "alerts", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_destination": { 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_destination", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email address to send alerts to", 2024-03-24T00:50:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_from_name": { 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_from_name", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "Ceph", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email From: name", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_host": { 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_host", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP server", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_password": { 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_password", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Password to authenticate with", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_port": { 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_port", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "465", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP port", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_sender": { 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_sender", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP envelope sender", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_ssl": { 2024-03-24T00:50:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_ssl", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_user": { 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_user", 2024-03-24T00:50:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "User to authenticate as", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "balancer", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "active": { 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "active", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_time": { 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_time", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0000", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_weekday": { 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_weekday", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_max_iterations": { 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_max_iterations", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "25", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "250", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_metrics": { 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_metrics", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_step": { 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_step", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.5", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0.001", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "0.999", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "aggressiveness of optimization", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "end_time": { 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_time", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2359", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "end_weekday": { 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_weekday", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "min_score": { 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_score", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "upmap", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-compat", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "none", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "read", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap-read" 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Balancer mode", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_ids": { 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_ids", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_deviation": { 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_deviation", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_optimizations": { 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_optimizations", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_down_multiplier": { 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_down_multiplier", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_refresh_rate": { 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_refresh_rate", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "20", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_starting_port": { 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_starting_port", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4721", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ptrace": { 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_ptrace", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_interval": { 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_interval", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_memory_target_ratio": { 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.7", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T00:50:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_log_destination": { 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm_log_destination", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "file", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "file,syslog", 2024-03-24T00:50:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "syslog" 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "cgroups_split": { 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cgroups_split", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "config_checks_enabled": { 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_checks_enabled", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "config_dashboard": { 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_dashboard", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_alertmanager": { 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_alertmanager", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_base": { 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_base", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Container image name, without the tag", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_elasticsearch": { 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_elasticsearch", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "elasticsearch container image", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_grafana": { 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_grafana", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_haproxy": { 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_haproxy", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "HAproxy container image", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_agent": { 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_agent", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger agent container image", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_collector": { 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_collector", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger collector container image", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_query": { 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_query", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger query container image", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_keepalived": { 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_keepalived", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Keepalived container image", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_loki": { 2024-03-24T00:50:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_loki", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Loki container image", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_node_exporter": { 2024-03-24T00:50:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_node_exporter", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nvmeof": { 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nvmeof", 2024-03-24T00:50:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nvme-of container image", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_prometheus": { 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_prometheus", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_promtail": { 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_promtail", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Promtail container image", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_snmp_gateway": { 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_snmp_gateway", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T00:50:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SNMP Gateway container image", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "container_init": { 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_init", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "daemon_cache_timeout": { 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "daemon_cache_timeout", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "default_cephadm_command_timeout": { 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "900", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "default_registry": { 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_registry", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "device_cache_timeout": { 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_cache_timeout", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1800", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache device inventory", 2024-03-24T00:50:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "device_enhanced_scan": { 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_enhanced_scan", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "facts_cache_timeout": { 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "facts_cache_timeout", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache host facts data", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "host_check_interval": { 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "host_check_interval", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to perform a host check", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "hw_monitoring": { 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hw_monitoring", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "inventory_list_all": { 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "inventory_list_all", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "log_refresh_metadata": { 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_refresh_metadata", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T00:50:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "*", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "max_count_per_host": { 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_count_per_host", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of daemons per service per host", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd_draining_count": { 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_osd_draining_count", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "migration_current": { 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "migration_current", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "internal - do not modify", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "root", 2024-03-24T00:50:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm-package", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "root" 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "oob_default_addr": { 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "oob_default_addr", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "169.254.1.1", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "prometheus_alerts_path": { 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus_alerts_path", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_insecure": { 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_insecure", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_password": { 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_password", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_url": { 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_url", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_username": { 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_username", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "secure_monitoring_stack": { 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secure_monitoring_stack", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "service_discovery_port": { 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "service_discovery_port", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8765", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "cephadm service discovery port", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_config_file": { 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_config_file", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "use_agent": { 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_agent", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "use_repo_digest": { 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_repo_digest", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_failed_host_check": { 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_failed_host_check", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_daemons": { 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_daemons", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_hosts": { 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_hosts", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crash", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "retain_interval": { 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retain_interval", 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "31536000", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_recent_interval": { 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_recent_interval", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1209600", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dashboard", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T00:50:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_ENABLED": { 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_URL": { 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_URL", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "{}", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "45", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T00:50:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T00:50:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "cross_origin_url": { 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cross_origin_url", 2024-03-24T00:50:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "crt_file": { 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crt_file", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "debug": { 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "debug", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable/disable debug options", 2024-03-24T00:50:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "jwt_token_ttl": { 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "jwt_token_ttl", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "28800", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "motd": { 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "motd", 2024-03-24T00:50:09.348 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "The message of the day", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect_resolve_ip_addr": { 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T00:50:09.349 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:09.350 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.351 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8080", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl_server_port": { 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl_server_port", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.353 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8443", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.354 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "redirect", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect" 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.355 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "url_prefix": { 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url_prefix", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.356 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.357 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_monitoring": { 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_monitoring", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "monitor device health metrics", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.359 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.360 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.361 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.362 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mark_out_threshold": { 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mark_out_threshold", 2024-03-24T00:50:09.363 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2419200", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": { 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_name", 2024-03-24T00:50:09.364 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "device_health_metrics", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "retention_period": { 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retention_period", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.365 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15552000", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain device health metrics", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_frequency": { 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_frequency", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.366 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "self_heal": { 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "self_heal", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.367 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.368 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_threshold": { 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_threshold", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7257600", 2024-03-24T00:50:09.369 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "diskprediction_local", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.370 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.371 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.372 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.373 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.374 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "predict_interval": { 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predict_interval", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.375 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "predictor_model": { 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predictor_model", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.376 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "prophetstor", 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.377 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "influx", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": false, 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "influxdb python module not found", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "batch_size": { 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "batch_size", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5000", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.379 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "database": { 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "database", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "ceph", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T00:50:09.380 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": { 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hostname", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server hostname", 2024-03-24T00:50:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "30", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "5", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.382 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.383 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.384 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.385 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.386 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.387 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "password": { 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "password", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "password of InfluxDB server user", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.388 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "port": { 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "port", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8086", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server port", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.389 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "false", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.390 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "threads": { 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threads", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "32", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.391 INFO:teuthology.orchestra.run.smithi123.stdout: "username": { 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "username", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "username of InfluxDB server user", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "verify_ssl": { 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "verify_ssl", 2024-03-24T00:50:09.392 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "true", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "insights", 2024-03-24T00:50:09.393 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.394 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.395 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.396 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.397 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "iostat", 2024-03-24T00:50:09.398 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.399 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.400 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.401 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.402 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.403 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "k8sevents", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_event_retention_days": { 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ceph_event_retention_days", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.404 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "config_check_secs": { 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_check_secs", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "10", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.405 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.406 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.407 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.408 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.409 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.410 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "localpool", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "failure_domain": { 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "failure_domain", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "host", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.411 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "failure domain for any created local pool", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.415 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": { 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_size", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default min_size for any created local pool", 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "num_rep": { 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "num_rep", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default replica count for any created local pool", 2024-03-24T00:50:09.418 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": { 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_num", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "128", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "prefix": { 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prefix", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name prefix for any created local pool", 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "subtree": { 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subtree", 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "rack", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mds_autoscaler", 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.424 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.426 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.427 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.428 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirroring", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.429 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.430 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.431 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.432 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.434 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "nfs", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.435 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.436 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.437 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.438 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.439 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "fail_fs": { 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fail_fs", 2024-03-24T00:50:09.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.441 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.442 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.443 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.444 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.445 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator": { 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.446 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rook", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "test_orchestrator" 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Orchestrator backend", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_perf_query", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.448 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.449 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.450 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.451 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.452 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_support", 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.453 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.454 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_autoscaler", 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.458 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.463 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "threshold": { 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threshold", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1.0", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "scaling threshold", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "progress", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_pg_recovery_event": { 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_pg_recovery_event", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.466 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.467 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.468 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.469 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.470 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.471 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.472 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completed_events": { 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_completed_events", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "50", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "number of past completed events to remember", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-03-24T00:50:09.473 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long the module is going to sleep", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "cache": { 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cache", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.475 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "exclude_perf_counters": { 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "exclude_perf_counters", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T00:50:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.477 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.478 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.479 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.480 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools": { 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools", 2024-03-24T00:50:09.481 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.482 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "300", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_interval": { 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_interval", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.483 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15.0", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.484 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "9283", 2024-03-24T00:50:09.485 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "stale_cache_strategy": { 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stale_cache_strategy", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "log", 2024-03-24T00:50:09.486 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "default", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.487 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "default", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "error" 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.488 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-03-24T00:50:09.489 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.490 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.491 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.492 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.493 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.494 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_snap_create": { 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_snap_create", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.495 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "mirror_snapshot_schedule": { 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.496 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "trash_purge_schedule": { 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "trash_purge_schedule", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.497 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "restful", 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_auth": { 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_auth", 2024-03-24T00:50:09.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.499 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.500 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.501 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.502 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.505 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.506 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rgw", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.507 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.508 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.509 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.510 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.511 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rook", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.512 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.513 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.514 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.515 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.516 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "storage_class": { 2024-03-24T00:50:09.517 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "storage_class", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "local", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "selftest", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.519 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.520 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.521 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.522 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.525 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "roption1": { 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption1", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.526 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "roption2": { 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption2", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.527 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "xyz", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption1": { 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption1", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption2": { 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption2", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.529 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption3": { 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption3", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.530 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption4": { 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption4", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.531 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption5": { 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption5", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.532 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption6": { 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption6", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.533 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption7": { 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption7", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.534 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "42", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "testkey": { 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testkey", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "testlkey": { 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testlkey", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.536 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "testnewline": { 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testnewline", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.537 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snap_schedule", 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_m_granularity": { 2024-03-24T00:50:09.538 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_m_granularity", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "dump_on_update": { 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dump_on_update", 2024-03-24T00:50:09.539 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "dump database to debug log on update", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.542 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.543 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.544 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stats", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.545 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.546 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.548 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.549 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "status", 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.550 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.551 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.552 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.553 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telegraf", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "address": { 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "address", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.556 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.557 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.558 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.559 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.560 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.561 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.562 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telemetry", 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_basic": { 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_basic", 2024-03-24T00:50:09.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_crash": { 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_crash", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.564 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_device": { 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_device", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.565 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_ident": { 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_ident", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.566 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_perf": { 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_perf", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "contact": { 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "contact", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.568 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "description": { 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "description", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.569 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "device_url": { 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_url", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T00:50:09.570 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.571 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "24", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "8", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.572 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "last_opt_revision": { 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "last_opt_revision", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.573 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard": { 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.574 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard_description": { 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard_description", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.575 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.576 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.577 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.578 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.580 INFO:teuthology.orchestra.run.smithi123.stdout: "organization": { 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "organization", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "proxy": { 2024-03-24T00:50:09.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "proxy", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "url": { 2024-03-24T00:50:09.582 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "test_orchestrator", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.584 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.586 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.587 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.589 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.591 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.592 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_clones": { 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_clones", 2024-03-24T00:50:09.593 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "periodic_async_work": { 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "periodic_async_work", 2024-03-24T00:50:09.594 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Periodically check for async work", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_delay": { 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_delay", 2024-03-24T00:50:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_no_wait": { 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.596 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix", 2024-03-24T00:50:09.597 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "discovery_interval": { 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "discovery_interval", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "100", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "identifier": { 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "identifier", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.599 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.600 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.601 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.602 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-03-24T00:50:09.603 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.604 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_host": { 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_host", 2024-03-24T00:50:09.605 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_port": { 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_port", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-03-24T00:50:09.606 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10051", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_sender": { 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_sender", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-03-24T00:50:09.607 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "services": { 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: "dashboard": "https://172.21.15.123:8443/" 2024-03-24T00:50:09.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "always_on_modules": { 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "octopus": [ 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "pacific": [ 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-03-24T00:50:09.609 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "quincy": [ 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-03-24T00:50:09.610 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "reef": [ 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-03-24T00:50:09.611 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "squid": [ 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-03-24T00:50:09.612 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "last_failure_osd_epoch": 3, 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "active_clients": [ 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:0", 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3644708448 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "libcephsqlite", 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:0", 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 126021721 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-03-24T00:50:09.614 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:0", 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2581123664 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:0", 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 197627345 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-24T00:50:09.615 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:08 smithi177 ceph-mon[36034]: pgmap v752: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:08 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3529444769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:08 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1578665376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:08 smithi204 ceph-mon[36345]: pgmap v752: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:08 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3529444769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:09.617 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:08 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1578665376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T00:50:09.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 msgr2=0x7fca100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 0x7fca100a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fca1802c8e0 tx=0x7fca1802df90 comp rx=0 tx=0).stop 2024-03-24T00:50:09.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 msgr2=0x7fca2c1999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c1999b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9c0 tx=0x7fca1c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 shutdown_connections 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.321+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca2c1052a0 0x7fca2c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca2c1048e0 0x7fca2c199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 --2- 172.21.15.123:0/1578665376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca2c1036f0 0x7fca2c1999b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 >> 172.21.15.123:0/1578665376 conn(0x7fca2c0fec80 msgr2=0x7fca2c107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 shutdown_connections 2024-03-24T00:50:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:08.322+0000 7fca31200640 1 -- 172.21.15.123:0/1578665376 wait complete. 2024-03-24T00:50:09.625 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T00:50:09.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T00:50:09.626 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:08 smithi123 ceph-mon[31487]: pgmap v752: 97 pgs: 97 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T00:50:09.626 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:08 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3529444769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:09.626 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:08 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1578665376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T00:50:09.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:09 smithi204 ceph-mon[36345]: pgmap v753: 97 pgs: 97 active+clean; 8.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T00:50:09.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:09 smithi123 ceph-mon[31487]: pgmap v753: 97 pgs: 97 active+clean; 8.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T00:50:09.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:09 smithi177 ceph-mon[36034]: pgmap v753: 97 pgs: 97 active+clean; 8.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T00:50:10.063 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 886, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 888, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 75765408, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 281758, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 8, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 4, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.368891195, 2024-03-24T00:50:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.052622226, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.249502476, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.048531910, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 886, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.097504850, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000110050 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 354, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 534, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 88971030, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 510752, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 7, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.377527114, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.041671335, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.305231948, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.021987827, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 532, 2024-03-24T00:50:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.076941551, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000144626 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 398432 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 727, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 723, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 46756845, 2024-03-24T00:50:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 295559, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 5, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 4, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.303448602, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.046383070, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.216348789, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.038687250, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 723, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.109059976, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000150843 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 498848, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 144672 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2841, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.158204330, 2024-03-24T00:50:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000055686 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 0, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 0, 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-03-24T00:50:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 954, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 574, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 380, 2024-03-24T00:50:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 134, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 2, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-03-24T00:50:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 4, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 234, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:50:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 0, 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 0 2024-03-24T00:50:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-03-24T00:50:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 0, 2024-03-24T00:50:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 99598, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 3, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 29, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 214044903, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 214703565, 2024-03-24T00:50:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 214699264, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 99598 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 21, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 893, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 872, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 16, 2024-03-24T00:50:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 126, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 110, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 954, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 1539, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 585, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 52952, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 215420 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 0, 2024-03-24T00:50:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 0, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 0, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-03-24T00:50:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-03-24T00:50:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-03-24T00:50:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-03-24T00:50:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 1, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 2, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 1, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 1, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 0, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 0, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 1986, 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-03-24T00:50:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 239431, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 78, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 7528, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 158, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 737658, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 4083, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 1713, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 26, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 4, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 1713, 2024-03-24T00:50:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1713, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 423.819141020, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.247413392 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1713, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 1714 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 1713, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 1713, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 0, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-03-24T00:50:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 973, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 737, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 2, 2024-03-24T00:50:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-03-24T00:50:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 89, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 2, 2024-03-24T00:50:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 11, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 11, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 2, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 0 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "osd-slow-ops": { 2024-03-24T00:50:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 0 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 1963, 2024-03-24T00:50:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 211300804, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 1963, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1963, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 211300804, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 1713, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 210520129, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1713, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 210520129, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 1713, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 1713, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1713, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1713, 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x558177174780": { 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-03-24T00:50:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:12.585 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:12.586 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:12 smithi123 ceph-mon[31487]: pgmap v754: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:50:12.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:12 smithi204 ceph-mon[36345]: pgmap v754: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:50:12.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:12 smithi177 ceph-mon[36034]: pgmap v754: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:50:13.488 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-24T00:50:13.488 DEBUG:tasks.check_counter:Getting stats from d 2024-03-24T00:50:13.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-24T00:50:13.907 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:50:14.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:14 smithi204 ceph-mon[36345]: pgmap v755: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 8 op/s 2024-03-24T00:50:14.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:14 smithi123 ceph-mon[31487]: pgmap v755: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 8 op/s 2024-03-24T00:50:14.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:14 smithi177 ceph-mon[36034]: pgmap v755: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 8 op/s 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:16.074 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.003530469, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.000800127, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:16.075 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 247, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 244, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 484752, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 146960, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.073514248, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.033489913, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.027210659, 2024-03-24T00:50:16.076 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.010922768, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 243, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.031176168, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000128296 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 484752, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 146960 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.077 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 6, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 189, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 3312, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 188656, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.033205524, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.024832312, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.004858391, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 188, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.020340475, 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000108194 2024-03-24T00:50:16.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 188656 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 0, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:16.079 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:16.080 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:16.081 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 147937, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:16.082 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 2, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:16.083 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 249, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 449872, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 249, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 249, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 449872, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:16.084 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:16.085 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:16.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:16.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:16.085 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:16.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:16 smithi204 ceph-mon[36345]: pgmap v756: 97 pgs: 97 active+clean; 8.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:16.834 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:16 smithi123 ceph-mon[31487]: pgmap v756: 97 pgs: 97 active+clean; 8.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:16.895 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-24T00:50:16.895 DEBUG:tasks.check_counter:Getting stats from g 2024-03-24T00:50:16.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-24T00:50:16.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:16 smithi177 ceph-mon[36034]: pgmap v756: 97 pgs: 97 active+clean; 8.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:17.330 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:50:18.705 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:18 smithi123 ceph-mon[31487]: pgmap v757: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:18.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:18 smithi204 ceph-mon[36345]: pgmap v757: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:18.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:18 smithi177 ceph-mon[36034]: pgmap v757: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:19.539 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.004049827, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.000815472, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:19.540 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 183, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 160, 2024-03-24T00:50:19.541 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 109680, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 3, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.028609920, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.022912666, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.002424893, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 181, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.021270467, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000117516 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:19.542 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 109680 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 246, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 241, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 485760, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 144576, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.059115507, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.020646730, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.028978723, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.009551352, 2024-03-24T00:50:19.543 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 241, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.037415278, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000155250 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 485760, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 144576 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 0, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.544 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:19.545 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-03-24T00:50:19.546 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 148967, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 29, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 0, 2024-03-24T00:50:19.547 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 2, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 243, 2024-03-24T00:50:19.548 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 448558, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 243, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 243, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 448558, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:19.549 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:19.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:19 smithi123 ceph-mon[31487]: pgmap v758: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:19.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:19 smithi204 ceph-mon[36345]: pgmap v758: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:19.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:19 smithi177 ceph-mon[36034]: pgmap v758: 97 pgs: 97 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:20.370 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-24T00:50:20.370 DEBUG:tasks.check_counter:Getting stats from j 2024-03-24T00:50:20.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-24T00:50:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:50:22.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:22 smithi204 ceph-mon[36345]: pgmap v759: 97 pgs: 97 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:22.849 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:22 smithi123 ceph-mon[31487]: pgmap v759: 97 pgs: 97 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:22.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:22 smithi177 ceph-mon[36034]: pgmap v759: 97 pgs: 97 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:23.050 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 254, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 249, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 488560, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 149280, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.065042946, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.022923104, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.028546901, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.009737246, 2024-03-24T00:50:23.051 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 249, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.027835413, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000111788 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 488560, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 149280 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 2, 2024-03-24T00:50:23.052 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 189, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 320, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 188560, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.035107886, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.027459137, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.003858503, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 187, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.020388408, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000109028 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:23.053 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 188560 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.001100465, 2024-03-24T00:50:23.054 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.000911505, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:23.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:23.056 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:23.057 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 148951, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:23.058 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:23.059 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 2, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 252, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 450262, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 252, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-03-24T00:50:23.060 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 252, 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 450262, 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-24T00:50:23.061 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-24T00:50:23.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:23 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:23.829 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:23 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:23.886 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-24T00:50:23.887 DEBUG:tasks.check_counter:Getting stats from b 2024-03-24T00:50:23.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-24T00:50:23.911 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:23 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:24.157 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:50:24.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:24 smithi123 ceph-mon[31487]: pgmap v760: 97 pgs: 97 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:24.622 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:24 smithi177 ceph-mon[36034]: pgmap v760: 97 pgs: 97 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:24.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:24 smithi204 ceph-mon[36345]: pgmap v760: 97 pgs: 97 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:25.568 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:25.568 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:25.568 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:25 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:50:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:25 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:50:25.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:25.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:25.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:25 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' 2024-03-24T00:50:25.925 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 6294, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 6336, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 2309344, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 99909317, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 9, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.652227464, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.343713645, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.089655638, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.180257164, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 6334, 2024-03-24T00:50:25.926 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.871797225, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000137637 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4170, 2024-03-24T00:50:25.927 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 4360, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1796034, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 82495269, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 189, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.541068679, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.294547797, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.097909356, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.126076962, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4359, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.546911933, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000354877 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:25.928 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 181, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000019573, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000108 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 487248 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 105737, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 109529, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 42712256, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 72574981, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 18, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 4.918083136, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 2.224139478, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 4.226727946, 2024-03-24T00:50:25.929 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.170909805, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 109528, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 6.911005408, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000063098 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 178545, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.024245248, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000135 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 41706752, 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 47031280 2024-03-24T00:50:25.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 11387, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 31.433029610, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002760431 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-24T00:50:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1999, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.137943617, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000069006 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 5380, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 5380, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 5380, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 106.461746227, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.019788428 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-03-24T00:50:25.932 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 402, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 1, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 3, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 26, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 5, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 21, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 18, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 3, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 3, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 2, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 10840, 2024-03-24T00:50:25.933 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 6293, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 62, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 5022, 2024-03-24T00:50:25.934 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 4, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 234, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 5022, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 1223, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 17, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 1206, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 3799, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 718, 2024-03-24T00:50:25.935 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 3905, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 6, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 96789, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 95052, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 179, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 2783 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-03-24T00:50:25.936 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 933, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 872, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 61, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 17, 2024-03-24T00:50:25.937 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 4, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 99598, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 99291, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 99291, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 307, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 290, 2024-03-24T00:50:25.938 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 125, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 124, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 124, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 1, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 1, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 2, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 214044903, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 214703565, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3530, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 3224.554503091, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.913471530 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-03-24T00:50:25.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 23, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 912, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 889, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 18, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 128, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 110, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 26, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 1860, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 1834, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 3, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 1043, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 1040, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 82300, 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 231804 2024-03-24T00:50:25.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 7722, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 5380, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 16, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 772, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2.199316972, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002848856 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 18, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.006522653, 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000362369 2024-03-24T00:50:25.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 61, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.182313594, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002988747 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.942 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2934, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.337137803, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000455738 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-24T00:50:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000229625, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000114812 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.944 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 105, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.269534101, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002566991 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 267, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.085592482, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000320571 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 177, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.185204249, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001046351 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 113, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.637738106, 2024-03-24T00:50:25.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.005643700 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-03-24T00:50:25.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 99, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 44.945567488, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.453995631 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 7, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.025775161, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.003682165 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.001501615, 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000375403 2024-03-24T00:50:25.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 7, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.018422364, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002631766 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 13, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.025780040, 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001983080 2024-03-24T00:50:25.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 801, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 56.541109974, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.070588152 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-03-24T00:50:25.952 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 2, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 1, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 1, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 435, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 435, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 751, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:25.953 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:25.954 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:25.955 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 330625, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 122, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 1760, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 20, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 7528, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 158, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:25.956 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 94250, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 627, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 13, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 12563, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 212764937, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 37, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 12550, 2024-03-24T00:50:25.957 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 12550, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 7199.920151379, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.573698816 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 12, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 12563, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 15108 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 12563, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 5096, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 7467, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-03-24T00:50:25.958 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 290, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 735, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 57, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 3572, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 131, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 1, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 2292, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 5039, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 1474, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:50:25.959 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 6, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-03-24T00:50:25.960 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 89, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 3, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 12, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 12, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-03-24T00:50:25.961 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 1262, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 0, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 249 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 3 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.962 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 50, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 51, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 2, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 32, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 1620, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 516, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 356 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 116197, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 31971880, 2024-03-24T00:50:25.963 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 116197, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 116197, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 31971880, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-24T00:50:25.964 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 12563, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 212853009, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 12550, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 212853009, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 13, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 12563, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 12563, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 12550, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 12550, 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:25.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 872, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 88072, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 872, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:25.967 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 42, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 88072, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x558ab6198500": { 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 99598, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 210509261, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 99598, 2024-03-24T00:50:25.968 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3530, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 210509261, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:25.969 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-24T00:50:26.702 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:26 smithi177 ceph-mon[36034]: pgmap v761: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:26.728 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-03-24T00:50:26.728 DEBUG:tasks.check_counter:Getting stats from e 2024-03-24T00:50:26.729 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-24T00:50:26.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:26 smithi204 ceph-mon[36345]: pgmap v761: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:26.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:26 smithi123 ceph-mon[31487]: pgmap v761: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T00:50:27.030 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:50:27.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:27 smithi177 ceph-mon[36034]: pgmap v762: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:27.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:27 smithi204 ceph-mon[36345]: pgmap v762: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:27.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:27 smithi123 ceph-mon[31487]: pgmap v762: 97 pgs: 97 active+clean; 7.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 253, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 248, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 488288, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 148656, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.028204280, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.009587188, 2024-03-24T00:50:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.013606859, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.004778267, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 248, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.007116452, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000028695 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 488288, 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 148656 2024-03-24T00:50:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 189, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 320, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 188560, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.007753024, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.005604560, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000838589, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 187, 2024-03-24T00:50:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.007459950, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000039892 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 188560 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.000192927, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000122942, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:28.768 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:28.769 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:28.770 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:28.771 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 147938, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:28.772 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 251, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 450132, 2024-03-24T00:50:28.773 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 251, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 251, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 450132, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:28.774 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-24T00:50:29.552 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-03-24T00:50:29.552 DEBUG:tasks.check_counter:Getting stats from h 2024-03-24T00:50:29.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-24T00:50:29.798 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:50:30.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:30 smithi204 ceph-mon[36345]: pgmap v763: 97 pgs: 97 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:50:30.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:30 smithi123 ceph-mon[31487]: pgmap v763: 97 pgs: 97 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:50:30.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:30 smithi177 ceph-mon[36034]: pgmap v763: 97 pgs: 97 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:50:31.533 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-24T00:50:31.533 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 4, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 2992, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 448, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.001366636, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000030939, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000485029, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000666082, 2024-03-24T00:50:31.534 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000222027 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 248, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 245, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 485024, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 147584, 2024-03-24T00:50:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.026112581, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.009470666, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.011980843, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.004529606, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 244, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.006814928, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000027930 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 485024, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 147584 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 188, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 320, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 188384, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.007646805, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.005333323, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000940857, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:31.537 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 186, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.008114204, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000043624 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 188384 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-24T00:50:31.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:31.539 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:31.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 147898, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 27, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:31.541 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:31.542 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 250, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 450002, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 250, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 250, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 450002, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:31.544 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-24T00:50:32.268 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-03-24T00:50:32.268 DEBUG:tasks.check_counter:Getting stats from k 2024-03-24T00:50:32.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-24T00:50:32.509 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:32 smithi177 ceph-mon[36034]: pgmap v764: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:32.527 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:50:32.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:32 smithi204 ceph-mon[36345]: pgmap v764: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:32.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:32 smithi123 ceph-mon[31487]: pgmap v764: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 253, 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 250, 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 486384, 2024-03-24T00:50:34.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 150704, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.028464337, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.009810476, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.013324808, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.004797114, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 249, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.006962561, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000027962 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:34.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 486384, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 150704 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 194, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 480, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 267264, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.008080457, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.005699667, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000919947, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:34.217 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 192, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.008776084, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000045708 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 267264 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4, 2024-03-24T00:50:34.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 4, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 2992, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 448, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.000831857, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000039918, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000671947, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000711005, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000237001 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:34.219 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:34.220 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:34.221 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 147898, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 27, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-24T00:50:34.224 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 256, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 450679, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 256, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 256, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 450679, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-24T00:50:34.225 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-24T00:50:34.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:34 smithi204 ceph-mon[36345]: pgmap v765: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:34.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:34 smithi123 ceph-mon[31487]: pgmap v765: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:34.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:34 smithi177 ceph-mon[36034]: pgmap v765: 97 pgs: 97 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:35.019 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-24T00:50:35.019 DEBUG:tasks.check_counter:Getting stats from c 2024-03-24T00:50:35.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-24T00:50:35.262 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:50:36.684 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:36 smithi204 ceph-mon[36345]: pgmap v766: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:36.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:36 smithi123 ceph-mon[31487]: pgmap v766: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:36.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:36 smithi177 ceph-mon[36034]: pgmap v766: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 254, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 249, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 488560, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 149280, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.035962256, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.012899277, 2024-03-24T00:50:37.109 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.016260690, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005696948, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 249, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.010390339, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000041728 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 488560, 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 149280 2024-03-24T00:50:37.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 190, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 320, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 266560, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.010443305, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.007449283, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001158794, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 188, 2024-03-24T00:50:37.111 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.010810513, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000057502 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 266560 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:37.112 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.000384815, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000323189, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:37.113 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:37.114 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:37.115 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:37.116 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 147960, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-03-24T00:50:37.117 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-03-24T00:50:37.118 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 252, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 450262, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 252, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 252, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 450262, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:37.119 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-24T00:50:37.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:37 smithi204 ceph-mon[36345]: pgmap v767: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:37.832 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-03-24T00:50:37.832 DEBUG:tasks.check_counter:Getting stats from f 2024-03-24T00:50:37.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-24T00:50:37.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:37 smithi123 ceph-mon[31487]: pgmap v767: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:37.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:37 smithi177 ceph-mon[36034]: pgmap v767: 97 pgs: 97 active+clean; 7.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:38.078 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:50:39.763 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 4, 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 4, 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 2992, 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 448, 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.001371736, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000038567, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000566194, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 3, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000673118, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000224372 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 191, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 320, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 188912, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.010406529, 2024-03-24T00:50:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.007598681, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000936592, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 189, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.010981820, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000058104 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T00:50:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 188912 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 251, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 248, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 485840, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 149456, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.034431453, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.012943592, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.014952555, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005911280, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 247, 2024-03-24T00:50:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.011918176, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000048251 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 485840, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 149456 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-03-24T00:50:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:39.770 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:39.771 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:39.772 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:39.773 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 147938, 2024-03-24T00:50:39.773 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:39.773 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:39.773 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:39.774 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-03-24T00:50:39.774 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-03-24T00:50:39.774 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:39.775 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 253, 2024-03-24T00:50:39.776 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 450392, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 253, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 253, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 450392, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:39.777 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-24T00:50:40.432 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-03-24T00:50:40.432 DEBUG:tasks.check_counter:Getting stats from i 2024-03-24T00:50:40.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-24T00:50:40.676 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:50:40.798 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:40 smithi204 ceph-mon[36345]: pgmap v768: 97 pgs: 97 active+clean; 7.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:40.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:40 smithi123 ceph-mon[31487]: pgmap v768: 97 pgs: 97 active+clean; 7.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:40.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:40 smithi177 ceph-mon[36034]: pgmap v768: 97 pgs: 97 active+clean; 7.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 250, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 247, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 485568, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 148832, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.033711422, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.012189268, 2024-03-24T00:50:42.363 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.014225548, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005685035, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 246, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.009114321, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000037050 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 485568, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 148832 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.364 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 6, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 191, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 3312, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 189008, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.010157613, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.007231520, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001238716, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 190, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.009422921, 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000049594 2024-03-24T00:50:42.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 189008 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:42.366 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.000304589, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000238168, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:42.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:42.368 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:42.369 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:42.370 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 147938, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:42.371 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-03-24T00:50:42.372 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-03-24T00:50:42.372 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:42.372 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:42.373 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:42.373 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:42.373 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:42.373 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-03-24T00:50:42.373 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-03-24T00:50:42.374 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 252, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 450262, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 252, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-03-24T00:50:42.375 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 252, 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 450262, 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:42.376 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-24T00:50:42.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:42 smithi177 ceph-mon[36034]: pgmap v769: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:42.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:42 smithi204 ceph-mon[36345]: pgmap v769: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:42.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:42 smithi123 ceph-mon[31487]: pgmap v769: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:43.023 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.i 2024-03-24T00:50:43.023 DEBUG:tasks.check_counter:Getting stats from l 2024-03-24T00:50:43.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 082c21f6-e975-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-24T00:50:43.269 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:50:44.746 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:44 smithi204 ceph-mon[36345]: pgmap v770: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:44.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:44 smithi123 ceph-mon[31487]: pgmap v770: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:44.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:44 smithi177 ceph-mon[36034]: pgmap v770: 97 pgs: 97 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 261, 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 256, 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 490464, 2024-03-24T00:50:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 153648, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.036542368, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.013530294, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.015992965, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005590461, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 256, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.009634099, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000037633 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 490464, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 153648 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 3, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 197, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 480, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 267792, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.010837666, 2024-03-24T00:50:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.007453645, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001353880, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 195, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.012403383, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000063607 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 267792 2024-03-24T00:50:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.000221186, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000149824, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:50:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:50:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:50:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 147960, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 28, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:50:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-03-24T00:50:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 260, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 451199, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 260, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-03-24T00:50:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 260, 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 451199, 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-03-24T00:50:45.020 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-24T00:50:45.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:45 smithi123 ceph-mon[31487]: pgmap v771: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:45.612 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:45 smithi204 ceph-mon[36345]: pgmap v771: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:45.638 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-24T00:50:45.638 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T00:50:45.650 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T00:50:45.650 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:50:45.651 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:50:45.696 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-24T00:50:45.696 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:50:45.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:50:45.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:45 smithi177 ceph-mon[36034]: pgmap v771: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:46.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 -- 172.21.15.123:0/4282769045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad881015e0 msgr2=0x7fad88101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 --2- 172.21.15.123:0/4282769045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad881015e0 0x7fad88101a60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fad74009900 tx=0x7fad7402bc50 comp rx=0 tx=0).stop 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 -- 172.21.15.123:0/4282769045 shutdown_connections 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 --2- 172.21.15.123:0/4282769045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad88101fa0 0x7fad88108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 --2- 172.21.15.123:0/4282769045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad881015e0 0x7fad88101a60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 --2- 172.21.15.123:0/4282769045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881003f0 0x7fad88100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 -- 172.21.15.123:0/4282769045 >> 172.21.15.123:0/4282769045 conn(0x7fad880fb980 msgr2=0x7fad880fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 -- 172.21.15.123:0/4282769045 shutdown_connections 2024-03-24T00:50:46.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.009+0000 7fad8e200640 1 -- 172.21.15.123:0/4282769045 wait complete. 2024-03-24T00:50:46.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.010+0000 7fad8e200640 1 Processor -- start 2024-03-24T00:50:46.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.010+0000 7fad8e200640 1 -- start start 2024-03-24T00:50:46.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 0x7fad88199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 0x7fad881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8810c0d0 con 0x7fad881003f0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8810bf50 con 0x7fad88101fa0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8810c250 con 0x7fad881015e0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 0x7fad881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:44822/0 (socket says 172.21.15.123:44822) 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad87e00640 1 -- 172.21.15.123:0/3913748619 learned_addr learned my addr 172.21.15.123:0/3913748619 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8ce00640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 0x7fad88199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 -- 172.21.15.123:0/3913748619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 msgr2=0x7fad88199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 -- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 msgr2=0x7fad88199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 0x7fad88199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.011+0000 7fad8d800640 1 -- 172.21.15.123:0/3913748619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad881a1640 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad87e00640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad8ce00640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 0x7fad88199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad8d800640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 0x7fad881a0fd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad80009fd0 tx=0x7fad8000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8000ce20 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad80005020 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad881a1930 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80010720 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.012+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad881a1e80 con 0x7fad88101fa0 2024-03-24T00:50:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.013+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad881a2110 con 0x7fad88101fa0 2024-03-24T00:50:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.015+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fad80005540 con 0x7fad88101fa0 2024-03-24T00:50:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.016+0000 7fad85600640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 0x7fad700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.016+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fad800cf1f0 con 0x7fad88101fa0 2024-03-24T00:50:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.017+0000 7fad8ce00640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 0x7fad700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.017+0000 7fad8ce00640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 0x7fad700a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fad88101440 tx=0x7fad7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.020+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fad8009c240 con 0x7fad88101fa0 2024-03-24T00:50:46.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.237+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad881a22e0 con 0x7fad88101fa0 2024-03-24T00:50:46.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.239+0000 7fad85600640 1 -- 172.21.15.123:0/3913748619 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7fad8009bb40 con 0x7fad88101fa0 2024-03-24T00:50:46.240 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:46.240 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:50:46.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 msgr2=0x7fad700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 0x7fad700a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fad88101440 tx=0x7fad7c005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:46.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 msgr2=0x7fad881a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 0x7fad881a0fd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad80009fd0 tx=0x7fad8000ef10 comp rx=0 tx=0).stop 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 shutdown_connections 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad88101fa0 0x7fad881a0fd0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad881015e0 0x7fad88199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fad700a2880 0x7fad700a4d40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 --2- 172.21.15.123:0/3913748619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad881003f0 0x7fad88199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 >> 172.21.15.123:0/3913748619 conn(0x7fad880fb980 msgr2=0x7fad880fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 shutdown_connections 2024-03-24T00:50:46.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.241+0000 7fad8e200640 1 -- 172.21.15.123:0/3913748619 wait complete. 2024-03-24T00:50:46.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:25.617002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:30.852647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:32.287029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:50:46.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:50:46.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 -- 172.21.15.123:0/779394600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe250102790 msgr2=0x7fe250102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 --2- 172.21.15.123:0/779394600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe250102790 0x7fe250102c10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009900 tx=0x7fe23c02bc50 comp rx=0 tx=0).stop 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 -- 172.21.15.123:0/779394600 shutdown_connections 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 --2- 172.21.15.123:0/779394600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250103150 0x7fe2501099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 --2- 172.21.15.123:0/779394600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe250102790 0x7fe250102c10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 --2- 172.21.15.123:0/779394600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2501015a0 0x7fe2501019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 -- 172.21.15.123:0/779394600 >> 172.21.15.123:0/779394600 conn(0x7fe2500fcb10 msgr2=0x7fe2500fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 -- 172.21.15.123:0/779394600 shutdown_connections 2024-03-24T00:50:46.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.443+0000 7fe256200640 1 -- 172.21.15.123:0/779394600 wait complete. 2024-03-24T00:50:46.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 Processor -- start 2024-03-24T00:50:46.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 -- start start 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 0x7fe2501978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250102790 0x7fe250197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25010d250 con 0x7fe250102790 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25010d0d0 con 0x7fe250103150 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.444+0000 7fe256200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25010d3d0 con 0x7fe2501015a0 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe24fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250102790 0x7fe250197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe254e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 0x7fe2501978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39366/0 (socket says 172.21.15.123:39366) 2024-03-24T00:50:46.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe254e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 0x7fe2501978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:44848/0 (socket says 172.21.15.123:44848) 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 -- 172.21.15.123:0/3930814608 learned_addr learned my addr 172.21.15.123:0/3930814608 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 -- 172.21.15.123:0/3930814608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 msgr2=0x7fe2501978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 0x7fe2501978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 -- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250102790 msgr2=0x7fe250197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250102790 0x7fe250197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 -- 172.21.15.123:0/3930814608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25019f4e0 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.445+0000 7fe255800640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe24800efd0 tx=0x7fe24800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.446+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248003b70 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.446+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe248003d10 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.446+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25019f7d0 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.446+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2480109f0 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.446+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25019fcf0 con 0x7fe250103150 2024-03-24T00:50:46.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.447+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2501088c0 con 0x7fe250103150 2024-03-24T00:50:46.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.448+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe248004020 con 0x7fe250103150 2024-03-24T00:50:46.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.450+0000 7fe24d600640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.450+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe2480cee40 con 0x7fe250103150 2024-03-24T00:50:46.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.450+0000 7fe254e00640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.451+0000 7fe254e00640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 0x7fe2380a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe2501025f0 tx=0x7fe244005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.454+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe24809bef0 con 0x7fe250103150 2024-03-24T00:50:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.658+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2500622a0 con 0x7fe250103150 2024-03-24T00:50:46.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.658+0000 7fe24d600640 1 -- 172.21.15.123:0/3930814608 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe24809b7f0 con 0x7fe250103150 2024-03-24T00:50:46.659 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-24T00:50:46.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 msgr2=0x7fe2380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 0x7fe2380a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe2501025f0 tx=0x7fe244005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:46.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 msgr2=0x7fe25019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe24800efd0 tx=0x7fe24800c6a0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 shutdown_connections 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe250103150 0x7fe25019ee70 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe250102790 0x7fe250197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 --2- 172.21.15.123:0/3930814608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2501015a0 0x7fe2501978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 >> 172.21.15.123:0/3930814608 conn(0x7fe2500fcb10 msgr2=0x7fe250105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 shutdown_connections 2024-03-24T00:50:46.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.661+0000 7fe256200640 1 -- 172.21.15.123:0/3930814608 wait complete. 2024-03-24T00:50:46.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:50:46.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:46 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3913748619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 -- 172.21.15.123:0/3880214377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e54105490 msgr2=0x7f8e5410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 --2- 172.21.15.123:0/3880214377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e54105490 0x7f8e5410bd20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009be0 tx=0x7f8e4c02bef0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 -- 172.21.15.123:0/3880214377 shutdown_connections 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 --2- 172.21.15.123:0/3880214377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e54105490 0x7f8e5410bd20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 --2- 172.21.15.123:0/3880214377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54104ad0 0x7f8e54104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 --2- 172.21.15.123:0/3880214377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e541038e0 0x7f8e54103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 -- 172.21.15.123:0/3880214377 >> 172.21.15.123:0/3880214377 conn(0x7f8e540fee50 msgr2=0x7f8e54101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 -- 172.21.15.123:0/3880214377 shutdown_connections 2024-03-24T00:50:46.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.867+0000 7f8e5b800640 1 -- 172.21.15.123:0/3880214377 wait complete. 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.868+0000 7f8e5b800640 1 Processor -- start 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.868+0000 7f8e5b800640 1 -- start start 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.868+0000 7f8e5b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.868+0000 7f8e5b800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e54104ad0 0x7f8e5419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e5b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54105490 0x7f8e541a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e5b800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410f590 con 0x7f8e54104ad0 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e5b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410f410 con 0x7f8e541038e0 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e5b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410f710 con 0x7f8e54105490 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39396/0 (socket says 172.21.15.123:39396) 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 -- 172.21.15.123:0/493165057 learned_addr learned my addr 172.21.15.123:0/493165057 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e53e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e54104ad0 0x7f8e5419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e59800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54105490 0x7f8e541a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 -- 172.21.15.123:0/493165057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54105490 msgr2=0x7f8e541a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54105490 0x7f8e541a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 -- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e54104ad0 msgr2=0x7f8e5419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e54104ad0 0x7f8e5419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 -- 172.21.15.123:0/493165057 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e541a17d0 con 0x7f8e541038e0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.869+0000 7f8e58e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9a0 tx=0x7f8e4800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.870+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800cd80 con 0x7f8e541038e0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.870+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e48010910 con 0x7f8e541038e0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.870+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48018a30 con 0x7f8e541038e0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.870+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e541a1ac0 con 0x7f8e541038e0 2024-03-24T00:50:46.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.870+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e541a1fc0 con 0x7f8e541038e0 2024-03-24T00:50:46.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.871+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e54068490 con 0x7f8e541038e0 2024-03-24T00:50:46.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.873+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8e48010ac0 con 0x7f8e541038e0 2024-03-24T00:50:46.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.874+0000 7f8e51600640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 0x7f8e3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:46.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.874+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8e480cef20 con 0x7f8e541038e0 2024-03-24T00:50:46.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.875+0000 7f8e53e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 0x7f8e3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:46.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.875+0000 7f8e53e00640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 0x7f8e3c0a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8e5419b0c0 tx=0x7f8e40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:46.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:46.876+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e4809bf70 con 0x7f8e541038e0 2024-03-24T00:50:46.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:46 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3913748619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:47.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:46 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3913748619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:47.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.073+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e5402d070 con 0x7f8e541038e0 2024-03-24T00:50:47.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.073+0000 7f8e51600640 1 -- 172.21.15.123:0/493165057 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e4809b870 con 0x7f8e541038e0 2024-03-24T00:50:47.075 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.075+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 msgr2=0x7f8e3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.075+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 0x7f8e3c0a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8e5419b0c0 tx=0x7f8e40005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.075+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 msgr2=0x7f8e54199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.075+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9a0 tx=0x7f8e4800ee60 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 shutdown_connections 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e54105490 0x7f8e541a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f8e3c0a2800 0x7f8e3c0a4cc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e54104ad0 0x7f8e5419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 --2- 172.21.15.123:0/493165057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e541038e0 0x7f8e54199bb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 >> 172.21.15.123:0/493165057 conn(0x7f8e540fee50 msgr2=0x7f8e54107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 shutdown_connections 2024-03-24T00:50:47.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.076+0000 7f8e5b800640 1 -- 172.21.15.123:0/493165057 wait complete. 2024-03-24T00:50:47.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:50:47.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 -- 172.21.15.123:0/1822797480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638102130 msgr2=0x7f6638104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 --2- 172.21.15.123:0/1822797480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638102130 0x7f6638104520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6628009900 tx=0x7f662802bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 -- 172.21.15.123:0/1822797480 shutdown_connections 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 --2- 172.21.15.123:0/1822797480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638107390 0x7f66381097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 --2- 172.21.15.123:0/1822797480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638104a60 0x7f6638106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 --2- 172.21.15.123:0/1822797480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638102130 0x7f6638104520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 -- 172.21.15.123:0/1822797480 >> 172.21.15.123:0/1822797480 conn(0x7f66380fbb40 msgr2=0x7f66380fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 -- 172.21.15.123:0/1822797480 shutdown_connections 2024-03-24T00:50:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.277+0000 7f663d800640 1 -- 172.21.15.123:0/1822797480 wait complete. 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 Processor -- start 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 -- start start 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 0x7f66381000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638107390 0x7f6638100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663810d360 con 0x7f6638102130 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663810d1e0 con 0x7f6638104a60 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.278+0000 7f663d800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663810d4e0 con 0x7f6638107390 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39532/0 (socket says 172.21.15.123:39532) 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 0x7f66381000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636a00640 1 -- 172.21.15.123:0/1943212153 learned_addr learned my addr 172.21.15.123:0/1943212153 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:47.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6637400640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638107390 0x7f6638100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 -- 172.21.15.123:0/1943212153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638107390 msgr2=0x7f6638100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638107390 0x7f6638100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 -- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 msgr2=0x7f6638101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 -- 172.21.15.123:0/1943212153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638100eb0 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636a00640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.279+0000 7f6636000640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 0x7f66381000e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f662400e9c0 tx=0x7f662400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.280+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662400ccc0 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.280+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6624010910 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.280+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66381a1b30 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.280+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240188c0 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.280+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66381a20d0 con 0x7f6638104a60 2024-03-24T00:50:47.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.281+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66380fd630 con 0x7f6638104a60 2024-03-24T00:50:47.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.283+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6624010ac0 con 0x7f6638104a60 2024-03-24T00:50:47.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.284+0000 7f662f400640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 0x7f661c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.285+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66240cedc0 con 0x7f6638104a60 2024-03-24T00:50:47.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.285+0000 7f6636a00640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 0x7f661c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.285+0000 7f6636a00640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 0x7f661c0a4d40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f662802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:47.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.286+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f662409b7c0 con 0x7f6638104a60 2024-03-24T00:50:47.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.482+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f663802d2c0 con 0x7f6638104a60 2024-03-24T00:50:47.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.482+0000 7f662f400640 1 -- 172.21.15.123:0/1943212153 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f662409f860 con 0x7f6638104a60 2024-03-24T00:50:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-03-24T00:50:47.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 msgr2=0x7f661c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 0x7f661c0a4d40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f662802c880 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 msgr2=0x7f66381000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 0x7f66381000e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f662400e9c0 tx=0x7f662400ee80 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 shutdown_connections 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6638107390 0x7f6638100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6638104a60 0x7f66381000e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f661c0a2880 0x7f661c0a4d40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 --2- 172.21.15.123:0/1943212153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6638102130 0x7f6638101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 >> 172.21.15.123:0/1943212153 conn(0x7f66380fbb40 msgr2=0x7f6638102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 shutdown_connections 2024-03-24T00:50:47.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.485+0000 7f663d800640 1 -- 172.21.15.123:0/1943212153 wait complete. 2024-03-24T00:50:47.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:50:47.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 -- 172.21.15.123:0/956840423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74102790 msgr2=0x7f0a74102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 --2- 172.21.15.123:0/956840423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74102790 0x7f0a74102c10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009900 tx=0x7f0a5c02bc50 comp rx=0 tx=0).stop 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 -- 172.21.15.123:0/956840423 shutdown_connections 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 --2- 172.21.15.123:0/956840423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74103150 0x7f0a741099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 --2- 172.21.15.123:0/956840423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74102790 0x7f0a74102c10 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 --2- 172.21.15.123:0/956840423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a741019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 -- 172.21.15.123:0/956840423 >> 172.21.15.123:0/956840423 conn(0x7f0a740fcb10 msgr2=0x7f0a740fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 -- 172.21.15.123:0/956840423 shutdown_connections 2024-03-24T00:50:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.690+0000 7f0a73200640 1 -- 172.21.15.123:0/956840423 wait complete. 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.691+0000 7f0a73200640 1 Processor -- start 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.691+0000 7f0a73200640 1 -- start start 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74103150 0x7f0a741a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7410d270 con 0x7f0a74102790 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7410d0f0 con 0x7f0a74103150 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a73200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7410d3f0 con 0x7f0a741015a0 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:44888/0 (socket says 172.21.15.123:44888) 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39556/0 (socket says 172.21.15.123:39556) 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 -- 172.21.15.123:0/1189907088 learned_addr learned my addr 172.21.15.123:0/1189907088 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a72800640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74103150 0x7f0a741a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 -- 172.21.15.123:0/1189907088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74103150 msgr2=0x7f0a741a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74103150 0x7f0a741a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 -- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 msgr2=0x7f0a74199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a71e00640 1 -- 172.21.15.123:0/1189907088 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a741a16c0 con 0x7f0a741015a0 2024-03-24T00:50:47.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.692+0000 7f0a6be00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a71e00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e9c0 tx=0x7f0a6400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400cd80 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a640035f0 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a741a19b0 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64010690 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.693+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a741a1e80 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.694+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a740683b0 con 0x7f0a741015a0 2024-03-24T00:50:47.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.695+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a640037a0 con 0x7f0a741015a0 2024-03-24T00:50:47.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.697+0000 7f0a6aa00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 0x7f0a580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:47.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.697+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0a640cee10 con 0x7f0a741015a0 2024-03-24T00:50:47.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.697+0000 7f0a6be00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 0x7f0a580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:47.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.698+0000 7f0a6be00640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 0x7f0a580a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c260 tx=0x7f0a5c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:47.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.701+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a640d2030 con 0x7f0a741015a0 2024-03-24T00:50:47.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3930814608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:47.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/493165057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:47.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:47 smithi123 ceph-mon[31487]: pgmap v772: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:47.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:47 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1943212153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.920+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a741a27f0 con 0x7f0a741015a0 2024-03-24T00:50:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.921+0000 7f0a6aa00640 1 -- 172.21.15.123:0/1189907088 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+601 (secure 0 0 0) 0x7f0a6409be60 con 0x7f0a741015a0 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/4001225558 2024-03-25T00:27:40.925167+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1263190299 2024-03-25T00:27:40.925167+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/3441797972 2024-03-25T00:27:07.030268+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/3441797972 2024-03-25T00:27:07.030268+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1307084792 2024-03-25T00:27:40.925167+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1263190299 2024-03-25T00:27:40.925167+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2789972175 2024-03-25T00:27:40.925167+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/828976007 2024-03-25T00:27:07.030268+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1787926427 2024-03-25T00:27:07.030268+0000 2024-03-24T00:50:47.922 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1360125430 2024-03-25T00:27:07.030268+0000 2024-03-24T00:50:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:listed 10 entries 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 msgr2=0x7f0a580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 0x7f0a580a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c260 tx=0x7f0a5c000bc0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 msgr2=0x7f0a74199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e9c0 tx=0x7f0a6400ee80 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 shutdown_connections 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a74103150 0x7f0a741a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f0a580a2800 0x7f0a580a4cc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.923+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a74102790 0x7f0a74199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.924+0000 7f0a73200640 1 --2- 172.21.15.123:0/1189907088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a741015a0 0x7f0a74199aa0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.924+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 >> 172.21.15.123:0/1189907088 conn(0x7f0a740fcb10 msgr2=0x7f0a740738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.924+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 shutdown_connections 2024-03-24T00:50:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:47.924+0000 7f0a73200640 1 -- 172.21.15.123:0/1189907088 wait complete. 2024-03-24T00:50:47.943 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:50:47.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:50:47.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:50:47.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:47 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3930814608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:47.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:47 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/493165057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:47.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:47 smithi177 ceph-mon[36034]: pgmap v772: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:47.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:47 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1943212153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:48.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:47 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3930814608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:50:48.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:47 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/493165057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:50:48.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:47 smithi204 ceph-mon[36345]: pgmap v772: 97 pgs: 97 active+clean; 7.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:50:48.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:47 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1943212153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:50:48.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:48 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/1189907088' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:50:48.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:48 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/1189907088' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:50:49.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:48 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/1189907088' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:50:49.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:49 smithi123 ceph-mon[31487]: pgmap v773: 97 pgs: 97 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:49.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:49 smithi177 ceph-mon[36034]: pgmap v773: 97 pgs: 97 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:50.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:49 smithi204 ceph-mon[36345]: pgmap v773: 97 pgs: 97 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 9 op/s 2024-03-24T00:50:50.615 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:50:50.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:50:50.615 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:50:50.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:50:50.699 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-03-24T00:50:50.743 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:50:50.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:50:50.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:50:50.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:50:50.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-03-24T00:50:50.971 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T00:50:50.984 INFO:tasks.cephadm:Teardown begin 2024-03-24T00:50:50.984 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:50:50.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 -- 172.21.15.123:0/120149314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 msgr2=0x7fd5ec104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/120149314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec104360 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009860 tx=0x7fd5dc02bb70 comp rx=0 tx=0).stop 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 -- 172.21.15.123:0/120149314 shutdown_connections 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/120149314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 0x7fd5ec109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/120149314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/120149314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec104360 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 -- 172.21.15.123:0/120149314 >> 172.21.15.123:0/120149314 conn(0x7fd5ec0fb980 msgr2=0x7fd5ec0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 -- 172.21.15.123:0/120149314 shutdown_connections 2024-03-24T00:50:51.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.166+0000 7fd5ebe00640 1 -- 172.21.15.123:0/120149314 wait complete. 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.167+0000 7fd5ebe00640 1 Processor -- start 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.167+0000 7fd5ebe00640 1 -- start start 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 0x7fd5ec1a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10d180 con 0x7fd5ec1071d0 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10d000 con 0x7fd5ec1048a0 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ebe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10d300 con 0x7fd5ec101f70 2024-03-24T00:50:51.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5eaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5eaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:44906/0 (socket says 172.21.15.123:44906) 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5eaa00640 1 -- 172.21.15.123:0/2922460891 learned_addr learned my addr 172.21.15.123:0/2922460891 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5ea000640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.168+0000 7fd5eb400640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 0x7fd5ec1a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 -- 172.21.15.123:0/2922460891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 msgr2=0x7fd5ec199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 -- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 msgr2=0x7fd5ec1a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 0x7fd5ec1a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 -- 172.21.15.123:0/2922460891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec1a1650 con 0x7fd5ec1048a0 2024-03-24T00:50:51.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5ea000640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec199f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec069360 tx=0x7fd5d800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.169+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d800ccd0 con 0x7fd5ec1048a0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.170+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5d80035f0 con 0x7fd5ec1048a0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.170+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8010650 con 0x7fd5ec1048a0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.170+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec1a1940 con 0x7fd5ec1048a0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.170+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec1a1e40 con 0x7fd5ec1048a0 2024-03-24T00:50:51.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.171+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec108d20 con 0x7fd5ec1048a0 2024-03-24T00:50:51.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.172+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd5d8004020 con 0x7fd5ec1048a0 2024-03-24T00:50:51.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.175+0000 7fd5e3400640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 0x7fd5d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.175+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd5d80cec60 con 0x7fd5ec1048a0 2024-03-24T00:50:51.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.175+0000 7fd5eaa00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 0x7fd5d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.175+0000 7fd5eaa00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 0x7fd5d00a4d40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0095f0 tx=0x7fd5dc034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:51.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.183+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5d809bcb0 con 0x7fd5ec1048a0 2024-03-24T00:50:51.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.415+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5ec062340 con 0x7fd5ec1048a0 2024-03-24T00:50:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.416+0000 7fd5e3400640 1 -- 172.21.15.123:0/2922460891 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fd5d8009510 con 0x7fd5ec1048a0 2024-03-24T00:50:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:51.416 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:50:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 msgr2=0x7fd5d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 0x7fd5d00a4d40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0095f0 tx=0x7fd5dc034ab0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 msgr2=0x7fd5ec199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec199f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec069360 tx=0x7fd5d800ee50 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 shutdown_connections 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fd5d00a2880 0x7fd5d00a4d40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5ec1071d0 0x7fd5ec1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5ec1048a0 0x7fd5ec199f70 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 --2- 172.21.15.123:0/2922460891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5ec101f70 0x7fd5ec199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 >> 172.21.15.123:0/2922460891 conn(0x7fd5ec0fb980 msgr2=0x7fd5ec101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 shutdown_connections 2024-03-24T00:50:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.419+0000 7fd5ebe00640 1 -- 172.21.15.123:0/2922460891 wait complete. 2024-03-24T00:50:51.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 -- 172.21.15.123:0/3470213859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc101f70 msgr2=0x7fabcc104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 --2- 172.21.15.123:0/3470213859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc101f70 0x7fabcc104360 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009900 tx=0x7fabbc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 -- 172.21.15.123:0/3470213859 shutdown_connections 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 --2- 172.21.15.123:0/3470213859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc1071d0 0x7fabcc109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 --2- 172.21.15.123:0/3470213859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1048a0 0x7fabcc106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 --2- 172.21.15.123:0/3470213859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc101f70 0x7fabcc104360 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 -- 172.21.15.123:0/3470213859 >> 172.21.15.123:0/3470213859 conn(0x7fabcc0fb980 msgr2=0x7fabcc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:51.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 -- 172.21.15.123:0/3470213859 shutdown_connections 2024-03-24T00:50:51.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.629+0000 7fabd2800640 1 -- 172.21.15.123:0/3470213859 wait complete. 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 Processor -- start 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 -- start start 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc1048a0 0x7fabcc197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1071d0 0x7fabcc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc10d1d0 con 0x7fabcc1071d0 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc10d050 con 0x7fabcc101f70 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd2800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc10d350 con 0x7fabcc1048a0 2024-03-24T00:50:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc1048a0 0x7fabcc197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.630+0000 7fabd1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39452/0 (socket says 172.21.15.123:39452) 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 -- 172.21.15.123:0/2771437634 learned_addr learned my addr 172.21.15.123:0/2771437634 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1e00640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1071d0 0x7fabcc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 -- 172.21.15.123:0/2771437634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc1048a0 msgr2=0x7fabcc197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc1048a0 0x7fabcc197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 -- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1071d0 msgr2=0x7fabcc19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1071d0 0x7fabcc19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 -- 172.21.15.123:0/2771437634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc19f420 con 0x7fabcc101f70 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd1400640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000c00 tx=0x7fabbc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc043020 con 0x7fabcc101f70 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabbc0054e0 con 0x7fabcc101f70 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc19f6b0 con 0x7fabcc101f70 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.631+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc003c40 con 0x7fabcc101f70 2024-03-24T00:50:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.632+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc19fc00 con 0x7fabcc101f70 2024-03-24T00:50:51.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.633+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc0fd470 con 0x7fabcc101f70 2024-03-24T00:50:51.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.635+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fabbc046020 con 0x7fabcc101f70 2024-03-24T00:50:51.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.636+0000 7fabca000640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 0x7fabb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:51.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.636+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fabbc0f0af0 con 0x7fabcc101f70 2024-03-24T00:50:51.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.636+0000 7fabd0a00640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 0x7fabb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:51.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.637+0000 7fabd0a00640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 0x7fabb40a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fabc00095f0 tx=0x7fabc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:51.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.640+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fabbc04d020 con 0x7fabcc101f70 2024-03-24T00:50:51.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:51 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2922460891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:51.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:51 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2922460891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:51.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.882+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabcc02d070 con 0x7fabcc101f70 2024-03-24T00:50:51.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.883+0000 7fabca000640 1 -- 172.21.15.123:0/2771437634 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11349 (secure 0 0 0) 0x7fabbc0bdb40 con 0x7fabcc101f70 2024-03-24T00:50:51.883 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:51.883 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2207292600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":2207292600},{"type":"v1","addr":"172.21.15.123:6837","nonce":2207292600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3661966001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3661966001},{"type":"v1","addr":"172.21.15.177:6833","nonce":3661966001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/812925941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":812925941},{"type":"v1","addr":"172.21.15.204:6833","nonce":812925941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24467,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/548064251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":548064251},{"type":"v1","addr":"172.21.15.123:6835","nonce":548064251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1091560059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1091560059},{"type":"v1","addr":"172.21.15.177:6835","nonce":1091560059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1788804579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1788804579},{"type":"v1","addr":"172.21.15.204:6835","nonce":1788804579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1145111503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1145111503},{"type":"v1","addr":"172.21.15.204:6837","nonce":1145111503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3793891487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3793891487},{"type":"v1","addr":"172.21.15.177:6837","nonce":3793891487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3993860651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3993860651},{"type":"v1","addr":"172.21.15.204:6839","nonce":3993860651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1841887169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1841887169},{"type":"v1","addr":"172.21.15.123:6839","nonce":1841887169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:35:32.796117+0000","modified":"2024-03-24T00:35:36.513248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6841/829451039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6840","nonce":829451039},{"type":"v1","addr":"172.21.15.123:6841","nonce":829451039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2781882004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2781882004},{"type":"v1","addr":"172.21.15.177:6839","nonce":2781882004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-24T00:50:51.884 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-24T00:50:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 msgr2=0x7fabb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 0x7fabb40a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fabc00095f0 tx=0x7fabc0005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 msgr2=0x7fabcc197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000c00 tx=0x7fabbc02c650 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 shutdown_connections 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fabb40a2800 0x7fabb40a4cc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1071d0 0x7fabcc19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fabcc1048a0 0x7fabcc197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 --2- 172.21.15.123:0/2771437634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc101f70 0x7fabcc197800 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.885+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 >> 172.21.15.123:0/2771437634 conn(0x7fabcc0fb980 msgr2=0x7fabcc102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.886+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 shutdown_connections 2024-03-24T00:50:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:51.886+0000 7fabd2800640 1 -- 172.21.15.123:0/2771437634 wait complete. 2024-03-24T00:50:51.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:50:51.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:51 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2922460891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 -- 172.21.15.123:0/1676168560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f81016b0 msgr2=0x7f42f8101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 --2- 172.21.15.123:0/1676168560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f81016b0 0x7f42f8101ad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f42ec009900 tx=0x7f42ec02bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:52.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 -- 172.21.15.123:0/1676168560 shutdown_connections 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 --2- 172.21.15.123:0/1676168560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f8103260 0x7f42f8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 --2- 172.21.15.123:0/1676168560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 --2- 172.21.15.123:0/1676168560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f81016b0 0x7f42f8101ad0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 -- 172.21.15.123:0/1676168560 >> 172.21.15.123:0/1676168560 conn(0x7f42f80fcc80 msgr2=0x7f42f80ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 -- 172.21.15.123:0/1676168560 shutdown_connections 2024-03-24T00:50:52.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.091+0000 7f4300800640 1 -- 172.21.15.123:0/1676168560 wait complete. 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.092+0000 7f4300800640 1 Processor -- start 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.092+0000 7f4300800640 1 -- start start 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 0x7f42f8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f8103260 0x7f42f8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f810d350 con 0x7f42f81016b0 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f810d1d0 con 0x7f42f81028a0 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f4300800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f810d4d0 con 0x7f42f8103260 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 0x7f42f8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39468/0 (socket says 172.21.15.123:39468) 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 0x7f42f8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39622/0 (socket says 172.21.15.123:39622) 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 -- 172.21.15.123:0/3264875963 learned_addr learned my addr 172.21.15.123:0/3264875963 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fe800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f8103260 0x7f42f8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 -- 172.21.15.123:0/3264875963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f8103260 msgr2=0x7f42f8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f8103260 0x7f42f8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 -- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 msgr2=0x7f42f8075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 0x7f42f8075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.093+0000 7f42fd400640 1 -- 172.21.15.123:0/3264875963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8074860 con 0x7f42f81028a0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f42fd400640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e9f0 tx=0x7f42e800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e800ccc0 con 0x7f42f81028a0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42e8010910 con 0x7f42f81028a0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f81a1b10 con 0x7f42f81028a0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e80188a0 con 0x7f42f81028a0 2024-03-24T00:50:52.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.094+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f81a2060 con 0x7f42f81028a0 2024-03-24T00:50:52.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.095+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f81089d0 con 0x7f42f81028a0 2024-03-24T00:50:52.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.097+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f42e8010ac0 con 0x7f42f81028a0 2024-03-24T00:50:52.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.098+0000 7f42f2a00640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 0x7f42e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.099+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f42e80cee90 con 0x7f42f81028a0 2024-03-24T00:50:52.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.099+0000 7f42fde00640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 0x7f42e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.099+0000 7f42fde00640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 0x7f42e00a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f42ec02c170 tx=0x7f42ec02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:52.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.102+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42e809bf40 con 0x7f42f81028a0 2024-03-24T00:50:52.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.298+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f80622a0 con 0x7f42f81028a0 2024-03-24T00:50:52.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.300+0000 7f42f2a00640 1 -- 172.21.15.123:0/3264875963 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22711 (secure 0 0 0) 0x7f42e809b840 con 0x7f42f81028a0 2024-03-24T00:50:52.301 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-24T00:50:52.301 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"082c21f6-e975-11ee-95cd-87774f69a715","created":"2024-03-24T00:26:34.341313+0000","modified":"2024-03-24T00:35:42.418066+0000","last_up_change":"2024-03-24T00:34:33.666496+0000","last_in_change":"2024-03-24T00:34:13.169394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:25.617002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:30.852647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:32.287029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfeabab-d57d-4786-9124-62355d56818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6803","nonce":407676320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6805","nonce":407676320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6809","nonce":407676320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":407676320},{"type":"v1","addr":"172.21.15.123:6807","nonce":407676320}]},"public_addr":"172.21.15.123:6803/407676320","cluster_addr":"172.21.15.123:6805/407676320","heartbeat_back_addr":"172.21.15.123:6809/407676320","heartbeat_front_addr":"172.21.15.123:6807/407676320","state":["exists","up"]},{"osd":1,"uuid":"97ec847a-50c8-46e1-bee7-f7ee7a409894","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6801","nonce":1151434087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6803","nonce":1151434087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6807","nonce":1151434087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1151434087},{"type":"v1","addr":"172.21.15.177:6805","nonce":1151434087}]},"public_addr":"172.21.15.177:6801/1151434087","cluster_addr":"172.21.15.177:6803/1151434087","heartbeat_back_addr":"172.21.15.177:6807/1151434087","heartbeat_front_addr":"172.21.15.177:6805/1151434087","state":["exists","up"]},{"osd":2,"uuid":"408c94aa-43dc-4ca2-82bf-e7c8973d41d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6801","nonce":823607592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6803","nonce":823607592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6807","nonce":823607592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":823607592},{"type":"v1","addr":"172.21.15.204:6805","nonce":823607592}]},"public_addr":"172.21.15.204:6801/823607592","cluster_addr":"172.21.15.204:6803/823607592","heartbeat_back_addr":"172.21.15.204:6807/823607592","heartbeat_front_addr":"172.21.15.204:6805/823607592","state":["exists","up"]},{"osd":3,"uuid":"c8245aa9-2f8d-48bd-bad2-0b2e706438d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6811","nonce":1321829693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6813","nonce":1321829693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6817","nonce":1321829693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1321829693},{"type":"v1","addr":"172.21.15.123:6815","nonce":1321829693}]},"public_addr":"172.21.15.123:6811/1321829693","cluster_addr":"172.21.15.123:6813/1321829693","heartbeat_back_addr":"172.21.15.123:6817/1321829693","heartbeat_front_addr":"172.21.15.123:6815/1321829693","state":["exists","up"]},{"osd":4,"uuid":"26f71028-4c8f-4813-ba49-216b3919c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6809","nonce":2757515144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6811","nonce":2757515144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6815","nonce":2757515144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2757515144},{"type":"v1","addr":"172.21.15.177:6813","nonce":2757515144}]},"public_addr":"172.21.15.177:6809/2757515144","cluster_addr":"172.21.15.177:6811/2757515144","heartbeat_back_addr":"172.21.15.177:6815/2757515144","heartbeat_front_addr":"172.21.15.177:6813/2757515144","state":["exists","up"]},{"osd":5,"uuid":"34571dde-a4d1-4a14-adbc-55097581ca68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6809","nonce":635738302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6811","nonce":635738302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6815","nonce":635738302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":635738302},{"type":"v1","addr":"172.21.15.204:6813","nonce":635738302}]},"public_addr":"172.21.15.204:6809/635738302","cluster_addr":"172.21.15.204:6811/635738302","heartbeat_back_addr":"172.21.15.204:6815/635738302","heartbeat_front_addr":"172.21.15.204:6813/635738302","state":["exists","up"]},{"osd":6,"uuid":"2a965499-6acb-447a-be25-fe19655bf999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6819","nonce":3042863338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6821","nonce":3042863338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6825","nonce":3042863338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3042863338},{"type":"v1","addr":"172.21.15.123:6823","nonce":3042863338}]},"public_addr":"172.21.15.123:6819/3042863338","cluster_addr":"172.21.15.123:6821/3042863338","heartbeat_back_addr":"172.21.15.123:6825/3042863338","heartbeat_front_addr":"172.21.15.123:6823/3042863338","state":["exists","up"]},{"osd":7,"uuid":"9c47f489-d125-48a6-8013-9702d5bf03f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6817","nonce":2491926780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6819","nonce":2491926780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6823","nonce":2491926780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2491926780},{"type":"v1","addr":"172.21.15.177:6821","nonce":2491926780}]},"public_addr":"172.21.15.177:6817/2491926780","cluster_addr":"172.21.15.177:6819/2491926780","heartbeat_back_addr":"172.21.15.177:6823/2491926780","heartbeat_front_addr":"172.21.15.177:6821/2491926780","state":["exists","up"]},{"osd":8,"uuid":"e2a2e425-e6a4-40c3-86fb-23ab0ca5efc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6817","nonce":4011283541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6819","nonce":4011283541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6823","nonce":4011283541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4011283541},{"type":"v1","addr":"172.21.15.204:6821","nonce":4011283541}]},"public_addr":"172.21.15.204:6817/4011283541","cluster_addr":"172.21.15.204:6819/4011283541","heartbeat_back_addr":"172.21.15.204:6823/4011283541","heartbeat_front_addr":"172.21.15.204:6821/4011283541","state":["exists","up"]},{"osd":9,"uuid":"234e47dd-50a2-4522-8559-5cbf492d88d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6827","nonce":3491687003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6829","nonce":3491687003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6833","nonce":3491687003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3491687003},{"type":"v1","addr":"172.21.15.123:6831","nonce":3491687003}]},"public_addr":"172.21.15.123:6827/3491687003","cluster_addr":"172.21.15.123:6829/3491687003","heartbeat_back_addr":"172.21.15.123:6833/3491687003","heartbeat_front_addr":"172.21.15.123:6831/3491687003","state":["exists","up"]},{"osd":10,"uuid":"52997bce-ef96-4fbd-803e-2c505beda096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6825","nonce":2089800664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6827","nonce":2089800664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6831","nonce":2089800664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2089800664},{"type":"v1","addr":"172.21.15.177:6829","nonce":2089800664}]},"public_addr":"172.21.15.177:6825/2089800664","cluster_addr":"172.21.15.177:6827/2089800664","heartbeat_back_addr":"172.21.15.177:6831/2089800664","heartbeat_front_addr":"172.21.15.177:6829/2089800664","state":["exists","up"]},{"osd":11,"uuid":"233d66e6-e30c-427c-b37e-dda971c1b299","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6825","nonce":2066593187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6827","nonce":2066593187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6831","nonce":2066593187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2066593187},{"type":"v1","addr":"172.21.15.204:6829","nonce":2066593187}]},"public_addr":"172.21.15.204:6825/2066593187","cluster_addr":"172.21.15.204:6827/2066593187","heartbeat_back_addr":"172.21.15.204:6831/2066593187","heartbeat_front_addr":"172.21.15.204:6829/2066593187","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:34.740521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:29:59.962637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:23.005740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:54.800096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:20.240295+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:31:44.230488+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:16.534827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:32:42.350287+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:06.657508+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:33:40.101305+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:06.135455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:34:30.835843+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:0/4001225558":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6801/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:6800/3441797972":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1307084792":"2024-03-25T00:27:40.925167+0000","172.21.15.123:6800/1263190299":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/2789972175":"2024-03-25T00:27:40.925167+0000","172.21.15.123:0/828976007":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1787926427":"2024-03-25T00:27:07.030268+0000","172.21.15.123:0/1360125430":"2024-03-25T00:27:07.030268+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 msgr2=0x7f42e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 0x7f42e00a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f42ec02c170 tx=0x7f42ec02df90 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 msgr2=0x7f42f8073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e9f0 tx=0x7f42e800edf0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 shutdown_connections 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42f8103260 0x7f42f8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f81028a0 0x7f42f8073a90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f42e00a2800 0x7f42e00a4cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 --2- 172.21.15.123:0/3264875963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f81016b0 0x7f42f8075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 >> 172.21.15.123:0/3264875963 conn(0x7f42f80fcc80 msgr2=0x7f42f8105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:52.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 shutdown_connections 2024-03-24T00:50:52.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.303+0000 7f4300800640 1 -- 172.21.15.123:0/3264875963 wait complete. 2024-03-24T00:50:52.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 -- 172.21.15.123:0/828797473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4100ffe50 msgr2=0x7fe410100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 --2- 172.21.15.123:0/828797473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4100ffe50 0x7fe410100270 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe400009900 tx=0x7fe40002bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 -- 172.21.15.123:0/828797473 shutdown_connections 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 --2- 172.21.15.123:0/828797473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe410101150 0x7fe4101015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 --2- 172.21.15.123:0/828797473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4101007b0 0x7fe410100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 --2- 172.21.15.123:0/828797473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4100ffe50 0x7fe410100270 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 -- 172.21.15.123:0/828797473 >> 172.21.15.123:0/828797473 conn(0x7fe4100fba10 msgr2=0x7fe4100fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 -- 172.21.15.123:0/828797473 shutdown_connections 2024-03-24T00:50:52.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.521+0000 7fe415000640 1 -- 172.21.15.123:0/828797473 wait complete. 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 Processor -- start 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 -- start start 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4101007b0 0x7fe410199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe410101150 0x7fe4101a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41010f430 con 0x7fe4101007b0 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41010f2b0 con 0x7fe410101150 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe415000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41010f5b0 con 0x7fe4100ffe50 2024-03-24T00:50:52.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe40f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe40f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:44966/0 (socket says 172.21.15.123:44966) 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe40f400640 1 -- 172.21.15.123:0/90451680 learned_addr learned my addr 172.21.15.123:0/90451680 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.523+0000 7fe40fe00640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe410101150 0x7fe4101a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe407e00640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4101007b0 0x7fe410199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 -- 172.21.15.123:0/90451680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe410101150 msgr2=0x7fe4101a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe410101150 0x7fe4101a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 -- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4101007b0 msgr2=0x7fe410199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4101007b0 0x7fe410199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 -- 172.21.15.123:0/90451680 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4101a16c0 con 0x7fe4100ffe50 2024-03-24T00:50:52.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.524+0000 7fe40f400640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe400000c00 tx=0x7fe40002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:52.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.525+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe400043020 con 0x7fe4100ffe50 2024-03-24T00:50:52.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.525+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4000054e0 con 0x7fe4100ffe50 2024-03-24T00:50:52.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.525+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4101a1950 con 0x7fe4100ffe50 2024-03-24T00:50:52.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.525+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe400003c40 con 0x7fe4100ffe50 2024-03-24T00:50:52.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.525+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4101a1ea0 con 0x7fe4100ffe50 2024-03-24T00:50:52.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.526+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41010acd0 con 0x7fe4100ffe50 2024-03-24T00:50:52.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.527+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe400046020 con 0x7fe4100ffe50 2024-03-24T00:50:52.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.529+0000 7fe40cc00640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:52.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.529+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe4000f0b20 con 0x7fe4100ffe50 2024-03-24T00:50:52.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.529+0000 7fe407e00640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:52.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.530+0000 7fe407e00640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe41019afb0 tx=0x7fe3f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:52.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.533+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe40004d020 con 0x7fe4100ffe50 2024-03-24T00:50:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-mon[31487]: pgmap v774: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2771437634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/3264875963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.810+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe4100622a0 con 0x7fe4100ffe50 2024-03-24T00:50:52.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.818+0000 7fe40cc00640 1 -- 172.21.15.123:0/90451680 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe4000bdb70 con 0x7fe4100ffe50 2024-03-24T00:50:52.818 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:50:52.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 msgr2=0x7fe3f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe41019afb0 tx=0x7fe3f8005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:52.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 msgr2=0x7fe410199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:52.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe400000c00 tx=0x7fe40002c650 comp rx=0 tx=0).stop 2024-03-24T00:50:52.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:52 smithi204 ceph-mon[36345]: pgmap v774: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:52 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2771437634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:52 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/3264875963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 shutdown_connections 2024-03-24T00:50:52.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe410101150 0x7fe4101a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4101007b0 0x7fe410199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 --2- 172.21.15.123:0/90451680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4100ffe50 0x7fe410199aa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:52.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.821+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 >> 172.21.15.123:0/90451680 conn(0x7fe4100fba10 msgr2=0x7fe4100fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:52.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.822+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 shutdown_connections 2024-03-24T00:50:52.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:52.822+0000 7fe415000640 1 -- 172.21.15.123:0/90451680 wait complete. 2024-03-24T00:50:52.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:50:52.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-mon[36034]: pgmap v774: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-24T00:50:52.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2771437634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:50:52.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/3264875963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:50:53.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 -- 172.21.15.123:0/1435591350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd0104ad0 msgr2=0x7f7dd0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:53.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 --2- 172.21.15.123:0/1435591350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd0104ad0 0x7f7dd0104f50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009900 tx=0x7f7dbc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:50:53.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 -- 172.21.15.123:0/1435591350 shutdown_connections 2024-03-24T00:50:53.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 --2- 172.21.15.123:0/1435591350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 --2- 172.21.15.123:0/1435591350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd0104ad0 0x7f7dd0104f50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 --2- 172.21.15.123:0/1435591350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd01038e0 0x7f7dd0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 -- 172.21.15.123:0/1435591350 >> 172.21.15.123:0/1435591350 conn(0x7f7dd00fee50 msgr2=0x7f7dd0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:53.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 -- 172.21.15.123:0/1435591350 shutdown_connections 2024-03-24T00:50:53.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.020+0000 7f7dd6600640 1 -- 172.21.15.123:0/1435591350 wait complete. 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.021+0000 7f7dd6600640 1 Processor -- start 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.021+0000 7f7dd6600640 1 -- start start 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 0x7f7dd0199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd0104ad0 0x7f7dd019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd010f5b0 con 0x7f7dd0104ad0 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd010f430 con 0x7f7dd01038e0 2024-03-24T00:50:53.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dd6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd010f730 con 0x7f7dd0105490 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd0104ad0 0x7f7dd019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.123:45006/0 (socket says 172.21.15.123:45006) 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 -- 172.21.15.123:0/2434312723 learned_addr learned my addr 172.21.15.123:0/2434312723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcf400640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 0x7f7dd0199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 -- 172.21.15.123:0/2434312723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 msgr2=0x7f7dd0199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 0x7f7dd0199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 -- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd0104ad0 msgr2=0x7f7dd019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd0104ad0 0x7f7dd019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcfe00640 1 -- 172.21.15.123:0/2434312723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd01a17e0 con 0x7f7dd0105490 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.022+0000 7f7dcf400640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 0x7f7dd0199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dcfe00640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800eb10 tx=0x7f7dc800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8004b20 con 0x7f7dd0105490 2024-03-24T00:50:53.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dc8004cc0 con 0x7f7dd0105490 2024-03-24T00:50:53.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8010760 con 0x7f7dd0105490 2024-03-24T00:50:53.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd01a1ad0 con 0x7f7dd0105490 2024-03-24T00:50:53.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.023+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01a1fa0 con 0x7f7dd0105490 2024-03-24T00:50:53.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.024+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd010aba0 con 0x7f7dd0105490 2024-03-24T00:50:53.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.026+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7dc8005020 con 0x7f7dd0105490 2024-03-24T00:50:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.027+0000 7f7dc7e00640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 0x7f7db40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.027+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8297+0+0 (secure 0 0 0) 0x7f7dc80cf0f0 con 0x7f7dd0105490 2024-03-24T00:50:53.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.028+0000 7f7dcf400640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 0x7f7db40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:53.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.028+0000 7f7dcf400640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 0x7f7db40a4d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0104930 tx=0x7f7dc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:53.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.032+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7dc809c150 con 0x7f7dd0105490 2024-03-24T00:50:53.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a[31483]: 2024-03-24T00:50:52.811+0000 7fdd1e400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:50:53.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: ignoring --setuser ceph since I am not root 2024-03-24T00:50:53.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: ignoring --setgroup ceph since I am not root 2024-03-24T00:50:53.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-mds[73130]: -- 172.21.15.123:0/3343045807 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f847d1ba40 con 0x55f847ceec00 2024-03-24T00:50:53.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:50:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: starting mds.a at 2024-03-24T00:50:53.212 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: ignoring --setuser ceph since I am not root 2024-03-24T00:50:53.213 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: ignoring --setgroup ceph since I am not root 2024-03-24T00:50:53.213 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-mds[67003]: -- 172.21.15.177:0/3344087840 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555772cbfa40 con 0x555772c93000 2024-03-24T00:50:53.213 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:50:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: starting mds.b at 2024-03-24T00:50:53.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.270+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7dd00622d0 con 0x7f7dd0105490 2024-03-24T00:50:53.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/90451680' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:50:53.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:50:53.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T00:50:53.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:50:53.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-24T00:50:53.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2434312723' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:53.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:53 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:53.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.827+0000 7f7dc7e00640 1 -- 172.21.15.123:0/2434312723 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7dc809ba50 con 0x7f7dd0105490 2024-03-24T00:50:53.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 msgr2=0x7f7db40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:53.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 0x7f7db40a4d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0104930 tx=0x7f7dc0005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:53.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 msgr2=0x7f7dd01a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:53.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800eb10 tx=0x7f7dc800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 shutdown_connections 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dd0105490 0x7f7dd01a1170 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f7db40a2880 0x7f7db40a4d40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd0104ad0 0x7f7dd019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 --2- 172.21.15.123:0/2434312723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dd01038e0 0x7f7dd0199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 >> 172.21.15.123:0/2434312723 conn(0x7f7dd00fee50 msgr2=0x7f7dd0107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 shutdown_connections 2024-03-24T00:50:53.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:53.830+0000 7f7dd6600640 1 -- 172.21.15.123:0/2434312723 wait complete. 2024-03-24T00:50:53.845 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/90451680' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2434312723' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:53.846 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:53 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:53.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:50:53.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/90451680' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2434312723' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:53.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:53 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 -- 172.21.15.123:0/3363408611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 msgr2=0x7f7854100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:54.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 --2- 172.21.15.123:0/3363408611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854100390 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7844009ca0 tx=0x7f784402c100 comp rx=0 tx=0).stop 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 -- 172.21.15.123:0/3363408611 shutdown_connections 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 --2- 172.21.15.123:0/3363408611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854101270 0x7f78541016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 --2- 172.21.15.123:0/3363408611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78541008d0 0x7f7854100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 --2- 172.21.15.123:0/3363408611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854100390 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 -- 172.21.15.123:0/3363408611 >> 172.21.15.123:0/3363408611 conn(0x7f78540fbb30 msgr2=0x7f78540fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 -- 172.21.15.123:0/3363408611 shutdown_connections 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.037+0000 7f7859600640 1 -- 172.21.15.123:0/3363408611 wait complete. 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.038+0000 7f7859600640 1 Processor -- start 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 -- start start 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:54.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 0x7f785419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7854101270 0x7f78541a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785410f500 con 0x7f78541008d0 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785410f380 con 0x7f78540fff70 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7859600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785410f680 con 0x7f7854101270 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.123:39548/0 (socket says 172.21.15.123:39548) 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 -- 172.21.15.123:0/4250434403 learned_addr learned my addr 172.21.15.123:0/4250434403 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7853400640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7854101270 0x7f78541a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852000640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 0x7f785419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 -- 172.21.15.123:0/4250434403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7854101270 msgr2=0x7f78541a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7854101270 0x7f78541a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 -- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 msgr2=0x7f785419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:54.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 0x7f785419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852a00640 1 -- 172.21.15.123:0/4250434403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78541a17f0 con 0x7f78540fff70 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.039+0000 7f7852000640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 0x7f785419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f7852a00640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7844009dc0 tx=0x7f7844035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844039040 con 0x7f78540fff70 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7844044070 con 0x7f78540fff70 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78541a1a80 con 0x7f78540fff70 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78440044a0 con 0x7f78540fff70 2024-03-24T00:50:54.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.040+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78541a1f70 con 0x7f78540fff70 2024-03-24T00:50:54.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.041+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785410afa0 con 0x7f78540fff70 2024-03-24T00:50:54.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.043+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7844004640 con 0x7f78540fff70 2024-03-24T00:50:54.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.044+0000 7f784b400640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 0x7f78380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:54.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.044+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8297+0+0 (secure 0 0 0) 0x7f78440f14a0 con 0x7f78540fff70 2024-03-24T00:50:54.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.044+0000 7f7852000640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 0x7f78380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:54.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.045+0000 7f7852000640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 0x7f78380a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f785419b0e0 tx=0x7f7840005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:54.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.046+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f78440be5c0 con 0x7f78540fff70 2024-03-24T00:50:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.261+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f785402d070 con 0x7f78540fff70 2024-03-24T00:50:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: pgmap v776: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s rd, 102 B/s wr, 10 op/s 2024-03-24T00:50:54.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: Cluster is now healthy 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.177:6838/1651997836,v1:172.21.15.177:6839/1651997836] up:boot 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: mds.? [v2:172.21.15.123:6840/3815885255,v1:172.21.15.123:6841/3815885255] up:boot 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: fsmap 12 up:standby 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:50:54.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/4250434403' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:54 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.834+0000 7f784b400640 1 -- 172.21.15.123:0/4250434403 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f78440bdec0 con 0x7f78540fff70 2024-03-24T00:50:54.834 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:50:54.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 msgr2=0x7f78380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 0x7f78380a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f785419b0e0 tx=0x7f7840005f90 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 msgr2=0x7f7854199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7844009dc0 tx=0x7f7844035d90 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 shutdown_connections 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7854101270 0x7f78541a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7f78380a2800 0x7f78380a4cc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78541008d0 0x7f785419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 --2- 172.21.15.123:0/4250434403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78540fff70 0x7f7854199bd0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 >> 172.21.15.123:0/4250434403 conn(0x7f78540fbb30 msgr2=0x7f78540fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.841+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 shutdown_connections 2024-03-24T00:50:54.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:54.842+0000 7f7859600640 1 -- 172.21.15.123:0/4250434403 wait complete. 2024-03-24T00:50:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: pgmap v776: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s rd, 102 B/s wr, 10 op/s 2024-03-24T00:50:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:50:54.856 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: Cluster is now healthy 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.177:6838/1651997836,v1:172.21.15.177:6839/1651997836] up:boot 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: mds.? [v2:172.21.15.123:6840/3815885255,v1:172.21.15.123:6841/3815885255] up:boot 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: fsmap 12 up:standby 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/4250434403' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.857 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:54 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:50:54.962 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: pgmap v776: 97 pgs: 97 active+clean; 7.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s rd, 102 B/s wr, 10 op/s 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: Cluster is now healthy 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.177:6838/1651997836,v1:172.21.15.177:6839/1651997836] up:boot 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: mds.? [v2:172.21.15.123:6840/3815885255,v1:172.21.15.123:6841/3815885255] up:boot 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: fsmap 12 up:standby 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: from='mgr.14152 172.21.15.123:0/1175154019' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/4250434403' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:54.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:54 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:55.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 -- 172.21.15.123:0/3628587552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84104ad0 msgr2=0x7fab84104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:55.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 --2- 172.21.15.123:0/3628587552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84104ad0 0x7fab84104f50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009900 tx=0x7fab6c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:50:55.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 -- 172.21.15.123:0/3628587552 shutdown_connections 2024-03-24T00:50:55.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 --2- 172.21.15.123:0/3628587552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84105490 0x7fab8410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 --2- 172.21.15.123:0/3628587552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84104ad0 0x7fab84104f50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 --2- 172.21.15.123:0/3628587552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 -- 172.21.15.123:0/3628587552 >> 172.21.15.123:0/3628587552 conn(0x7fab840fee50 msgr2=0x7fab84101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 -- 172.21.15.123:0/3628587552 shutdown_connections 2024-03-24T00:50:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.060+0000 7fab89a00640 1 -- 172.21.15.123:0/3628587552 wait complete. 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.061+0000 7fab89a00640 1 Processor -- start 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.061+0000 7fab89a00640 1 -- start start 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84104ad0 0x7fab8419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84105490 0x7fab841a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410f5b0 con 0x7fab841038e0 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410f430 con 0x7fab84105490 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab89a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410f730 con 0x7fab84104ad0 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:55.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39730/0 (socket says 172.21.15.123:39730) 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 -- 172.21.15.123:0/2308611319 learned_addr learned my addr 172.21.15.123:0/2308611319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab83400640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84105490 0x7fab841a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab7a000640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84104ad0 0x7fab8419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 -- 172.21.15.123:0/2308611319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84104ad0 msgr2=0x7fab8419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84104ad0 0x7fab8419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 -- 172.21.15.123:0/2308611319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84105490 msgr2=0x7fab841a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84105490 0x7fab841a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.062+0000 7fab82a00640 1 -- 172.21.15.123:0/2308611319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab841a1870 con 0x7fab841038e0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab82a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab7400e9c0 tx=0x7fab7400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7400ccc0 con 0x7fab841038e0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab841a1b60 con 0x7fab841038e0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab740035f0 con 0x7fab841038e0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab841a2060 con 0x7fab841038e0 2024-03-24T00:50:55.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.063+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7400ce60 con 0x7fab841038e0 2024-03-24T00:50:55.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.064+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8410aca0 con 0x7fab841038e0 2024-03-24T00:50:55.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.066+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fab740037a0 con 0x7fab841038e0 2024-03-24T00:50:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.069+0000 7fab7be00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 0x7fab680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:55.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.069+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7743+0+0 (secure 0 0 0) 0x7fab740cf1d0 con 0x7fab841038e0 2024-03-24T00:50:55.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.069+0000 7fab7a000640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 0x7fab680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:55.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.070+0000 7fab7a000640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 0x7fab680a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02c250 tx=0x7fab6c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:55.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.072+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab7409c410 con 0x7fab841038e0 2024-03-24T00:50:55.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.270+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fab8402d070 con 0x7fab841038e0 2024-03-24T00:50:55.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.840+0000 7fab7be00640 1 -- 172.21.15.123:0/2308611319 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7fab7409bd10 con 0x7fab841038e0 2024-03-24T00:50:55.840 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-03-24T00:50:55.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.845+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 msgr2=0x7fab680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:55.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.845+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 0x7fab680a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02c250 tx=0x7fab6c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.845+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 msgr2=0x7fab84199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:55.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.845+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab7400e9c0 tx=0x7fab7400edc0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 shutdown_connections 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84105490 0x7fab841a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab84104ad0 0x7fab8419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:6800/2933502234,v1:172.21.15.123:6801/2933502234] conn(0x7fab680a2800 0x7fab680a4cc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 --2- 172.21.15.123:0/2308611319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab841038e0 0x7fab84199c60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 >> 172.21.15.123:0/2308611319 conn(0x7fab840fee50 msgr2=0x7fab84107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:55.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 shutdown_connections 2024-03-24T00:50:55.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-24T00:50:55.847+0000 7fab89a00640 1 -- 172.21.15.123:0/2308611319 wait complete. 2024-03-24T00:50:55.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:50:55.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:50:56.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:50:56.089 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T00:50:56.089 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:50:56.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:55 smithi123 ceph-mon[31487]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T00:50:56.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:55 smithi123 ceph-mon[31487]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:50:56.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:55 smithi123 ceph-mon[31487]: from='client.? 172.21.15.123:0/2308611319' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:56.107 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:55 smithi123 ceph-mon[31487]: pgmap v778: 65 pgs: 65 active+clean; 7.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:50:56.147 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:50:56.172 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:50:56.195 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T00:50:56.196 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T00:50:56.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a 2024-03-24T00:50:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:55 smithi177 ceph-mon[36034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T00:50:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:55 smithi177 ceph-mon[36034]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:50:56.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:55 smithi177 ceph-mon[36034]: from='client.? 172.21.15.123:0/2308611319' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:56.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:55 smithi177 ceph-mon[36034]: pgmap v778: 65 pgs: 65 active+clean; 7.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:50:56.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:55 smithi204 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T00:50:56.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:55 smithi204 ceph-mon[36345]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:50:56.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:55 smithi204 ceph-mon[36345]: from='client.? 172.21.15.123:0/2308611319' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T00:50:56.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:55 smithi204 ceph-mon[36345]: pgmap v778: 65 pgs: 65 active+clean; 7.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:50:56.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:56 smithi123 systemd[1]: Stopping Ceph mon.a for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:50:57.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a[31483]: 2024-03-24T00:50:56.812+0000 7fdd25200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:50:57.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a[31483]: 2024-03-24T00:50:56.812+0000 7fdd25200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T00:50:57.106 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:56 smithi123 podman[97391]: 2024-03-24 00:50:56.956294668 +0000 UTC m=+0.272100183 container died 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T00:50:57.606 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:57 smithi123 podman[97391]: 2024-03-24 00:50:57.407326957 +0000 UTC m=+0.723132469 container cleanup 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:50:57.607 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:57 smithi123 bash[97391]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a 2024-03-24T00:50:57.999 INFO:journalctl@ceph.mon.a.smithi123.stdout:Mar 24 00:50:57 smithi123 podman[97405]: 2024-03-24 00:50:57.658677223 +0000 UTC m=+0.684518309 container remove 1902959791ff408b492052e8cc6a577f3fa546ccacacc8b4c8f053be4fbb8acc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:50:58.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T00:50:58.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:50:58.142 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T00:50:58.142 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T00:50:58.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.b 2024-03-24T00:50:58.426 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 systemd[1]: Stopping Ceph mon.b for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:50:58.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b[36030]: 2024-03-24T00:50:58.426+0000 7fcd88b8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:50:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b[36030]: 2024-03-24T00:50:58.426+0000 7fcd88b8a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T00:50:58.713 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 podman[71526]: 2024-03-24 00:50:58.620186189 +0000 UTC m=+0.290247718 container died 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:50:59.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 podman[71526]: 2024-03-24 00:50:58.993799783 +0000 UTC m=+0.663861307 container cleanup 4d40ca76012ff177f23e33c0e895f4ae80172404c1b022a3aa2f34ad46de4430 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:50:59.213 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 24 00:50:58 smithi177 bash[71526]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-b 2024-03-24T00:50:59.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T00:50:59.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:50:59.407 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T00:50:59.407 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T00:50:59.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.c 2024-03-24T00:50:59.692 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:59 smithi204 systemd[1]: Stopping Ceph mon.c for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:00.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c[36339]: 2024-03-24T00:50:59.690+0000 7fd52ae4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:51:00.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c[36339]: 2024-03-24T00:50:59.690+0000 7fd52ae4a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T00:51:00.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:50:59 smithi204 podman[72333]: 2024-03-24 00:50:59.935885199 +0000 UTC m=+0.342387096 container died 75b28aac96f57f8091f37cda4fb743f2e6b5b9acb266dd27f166e80ca205ab8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T00:51:00.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:51:00 smithi204 podman[72333]: 2024-03-24 00:51:00.265939043 +0000 UTC m=+0.672440950 container cleanup 75b28aac96f57f8091f37cda4fb743f2e6b5b9acb266dd27f166e80ca205ab8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T00:51:00.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:51:00 smithi204 bash[72333]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c 2024-03-24T00:51:00.731 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 24 00:51:00 smithi204 podman[72347]: 2024-03-24 00:51:00.517969012 +0000 UTC m=+0.586215551 container remove 75b28aac96f57f8091f37cda4fb743f2e6b5b9acb266dd27f166e80ca205ab8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T00:51:00.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T00:51:00.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:51:00.799 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T00:51:00.799 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T00:51:00.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x 2024-03-24T00:51:01.336 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:00 smithi123 systemd[1]: Stopping Ceph mgr.x for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:01.607 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:51:01.335+0000 7f0c18a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:51:01.607 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x[31729]: 2024-03-24T00:51:01.335+0000 7f0c18a00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T00:51:01.607 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:01 smithi123 podman[97512]: 2024-03-24 00:51:01.437218032 +0000 UTC m=+0.248916115 container died 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T00:51:01.985 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:01 smithi123 podman[97512]: 2024-03-24 00:51:01.979516143 +0000 UTC m=+0.791214268 container cleanup 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:51:01.985 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:01 smithi123 bash[97512]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x 2024-03-24T00:51:02.606 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Mar 24 00:51:02 smithi123 podman[97526]: 2024-03-24 00:51:02.296917794 +0000 UTC m=+0.850728620 container remove 7328f12350232974277bbfbeafc5190bece19cf238f7bbb3e994fc41c838a900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-24T00:51:02.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T00:51:02.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:51:02.891 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T00:51:02.891 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T00:51:02.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.y 2024-03-24T00:51:03.210 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:51:02 smithi177 systemd[1]: Stopping Ceph mgr.y for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:03.462 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:51:03 smithi177 podman[71649]: 2024-03-24 00:51:03.210402616 +0000 UTC m=+0.138991219 container died 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T00:51:03.849 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:51:03 smithi177 podman[71649]: 2024-03-24 00:51:03.589953436 +0000 UTC m=+0.518542026 container cleanup 6a1dcfd5dcba5440458df0643e8e4f1a5e9d8ba1d5e1c0413c4dc36528e39026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T00:51:03.849 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Mar 24 00:51:03 smithi177 bash[71649]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-y 2024-03-24T00:51:04.009 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T00:51:04.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:51:04.045 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T00:51:04.046 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T00:51:04.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.z 2024-03-24T00:51:04.322 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:51:04 smithi204 systemd[1]: Stopping Ceph mgr.z for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:04.622 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:51:04 smithi204 podman[72467]: 2024-03-24 00:51:04.349874252 +0000 UTC m=+0.121977903 container died d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:51:04.988 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:51:04 smithi204 podman[72467]: 2024-03-24 00:51:04.729910064 +0000 UTC m=+0.502013707 container cleanup d16621cf933ee2e8f0b7bec128a3821a940d24de4873057be1dcb4920b535e02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T00:51:04.988 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 24 00:51:04 smithi204 bash[72467]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mgr-z 2024-03-24T00:51:05.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T00:51:05.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:51:05.197 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T00:51:05.197 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T00:51:05.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.0 2024-03-24T00:51:05.606 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:05 smithi123 systemd[1]: Stopping Ceph osd.0 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:06.107 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0[46414]: 2024-03-24T00:51:05.730+0000 7f3b91800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:51:06.107 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0[46414]: 2024-03-24T00:51:05.730+0000 7f3b91800700 -1 osd.0 92 *** Got signal Terminated *** 2024-03-24T00:51:06.107 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0[46414]: 2024-03-24T00:51:05.730+0000 7f3b91800700 -1 osd.0 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:51:36.108 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:35 smithi123 bash[97633]: time="2024-03-24T00:51:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0 in 30 seconds, resorting to SIGKILL" 2024-03-24T00:51:36.108 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:35 smithi123 podman[97633]: 2024-03-24 00:51:35.900942722 +0000 UTC m=+30.310139248 container died d28b733fb9796593501d953f18d5dd7b0bfb9e8634a72716b89f3b0319f3b1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:51:36.599 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:36 smithi123 podman[97633]: 2024-03-24 00:51:36.322986016 +0000 UTC m=+30.732182602 container cleanup d28b733fb9796593501d953f18d5dd7b0bfb9e8634a72716b89f3b0319f3b1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:51:36.599 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:36 smithi123 bash[97633]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0 2024-03-24T00:51:36.856 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:36 smithi123 podman[97664]: 2024-03-24 00:51:36.598790856 +0000 UTC m=+0.680299588 container remove d28b733fb9796593501d953f18d5dd7b0bfb9e8634a72716b89f3b0319f3b1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-24T00:51:36.856 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:36 smithi123 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.0.service: Main process exited, code=exited, status=137/n/a 2024-03-24T00:51:37.487 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.076951749 +0000 UTC m=+0.032362788 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:51:37.487 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.183040368 +0000 UTC m=+0.138451394 container create 6d1c1f58f90034d582102c4a12b93eea268b39b3040d84f40ddb8e66060e42c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T00:51:37.856 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.487352739 +0000 UTC m=+0.442763847 container init 6d1c1f58f90034d582102c4a12b93eea268b39b3040d84f40ddb8e66060e42c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T00:51:37.857 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.498039256 +0000 UTC m=+0.453450298 container start 6d1c1f58f90034d582102c4a12b93eea268b39b3040d84f40ddb8e66060e42c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T00:51:37.857 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.577306504 +0000 UTC m=+0.532717553 container attach 6d1c1f58f90034d582102c4a12b93eea268b39b3040d84f40ddb8e66060e42c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:51:38.209 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 24 00:51:37 smithi123 podman[97753]: 2024-03-24 00:51:37.923041349 +0000 UTC m=+0.878452411 container died 6d1c1f58f90034d582102c4a12b93eea268b39b3040d84f40ddb8e66060e42c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T00:51:38.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T00:51:38.707 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:51:38.707 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T00:51:38.707 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T00:51:38.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.1 2024-03-24T00:51:39.033 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:51:38 smithi177 systemd[1]: Stopping Ceph osd.1 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:51:39.462 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:51:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:51:39.032+0000 7f91107cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:51:39.463 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:51:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:51:39.032+0000 7f91107cc700 -1 osd.1 92 *** Got signal Terminated *** 2024-03-24T00:51:39.463 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:51:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1[40658]: 2024-03-24T00:51:39.032+0000 7f91107cc700 -1 osd.1 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:52:09.373 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 bash[71786]: time="2024-03-24T00:52:09Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1 in 30 seconds, resorting to SIGKILL" 2024-03-24T00:52:09.373 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 podman[71786]: 2024-03-24 00:52:09.100138325 +0000 UTC m=+30.204843202 container died 173905085e363bb1769b662dc1f37c469730c4f1f4e7f35ba4db5baed12b907c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:52:09.713 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 podman[71786]: 2024-03-24 00:52:09.471844799 +0000 UTC m=+30.576549689 container cleanup 173905085e363bb1769b662dc1f37c469730c4f1f4e7f35ba4db5baed12b907c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-24T00:52:09.713 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 bash[71786]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1 2024-03-24T00:52:10.213 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 podman[71804]: 2024-03-24 00:52:09.823273357 +0000 UTC m=+0.719246803 container remove 173905085e363bb1769b662dc1f37c469730c4f1f4e7f35ba4db5baed12b907c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:52:10.213 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:09 smithi177 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.1.service: Main process exited, code=exited, status=137/n/a 2024-03-24T00:52:10.543 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.127187957 +0000 UTC m=+0.017990199 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:52:10.543 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.241545776 +0000 UTC m=+0.132348040 container create c8e547d056e99c53196ece6b6da19b27d4c82b414628c09ad0333b2d0c58a96f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:52:10.544 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.469202605 +0000 UTC m=+0.360004834 container init c8e547d056e99c53196ece6b6da19b27d4c82b414628c09ad0333b2d0c58a96f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-24T00:52:10.544 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.472559431 +0000 UTC m=+0.363361660 container start c8e547d056e99c53196ece6b6da19b27d4c82b414628c09ad0333b2d0c58a96f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:52:10.544 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.543364039 +0000 UTC m=+0.434166282 container attach c8e547d056e99c53196ece6b6da19b27d4c82b414628c09ad0333b2d0c58a96f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T00:52:10.963 INFO:journalctl@ceph.osd.1.smithi177.stdout:Mar 24 00:52:10 smithi177 podman[71900]: 2024-03-24 00:52:10.766833231 +0000 UTC m=+0.657635447 container died c8e547d056e99c53196ece6b6da19b27d4c82b414628c09ad0333b2d0c58a96f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:52:11.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T00:52:11.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:52:11.374 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T00:52:11.375 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T00:52:11.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.2 2024-03-24T00:52:11.655 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:11 smithi204 systemd[1]: Stopping Ceph osd.2 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:52:11.926 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2[41552]: 2024-03-24T00:52:11.654+0000 7fbefc6d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:52:11.927 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2[41552]: 2024-03-24T00:52:11.654+0000 7fbefc6d5700 -1 osd.2 92 *** Got signal Terminated *** 2024-03-24T00:52:11.927 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2[41552]: 2024-03-24T00:52:11.654+0000 7fbefc6d5700 -1 osd.2 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:52:37.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:37.463+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:37.823 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:37.579+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:38.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:38 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:38.187+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:38.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:38.462+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:38.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:38.580+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:38.822 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:38.364+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:38.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:38.430+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:39.357 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:39.040+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:39.379 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:39.200+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:39.380 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:39.126+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:39.593 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:39.333+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:39.593 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:39.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:39.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:39.441+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:40.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:39.592+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:40.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:40.043+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:40.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:40 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:40.224+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:40.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:40 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:40.084+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:40.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:40.544+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:40.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:40.566+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:40.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:40.316+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:40.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:40.411+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:41.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:41.062+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:41.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:41 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:41.105+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:41.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:41 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:41.261+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:41.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:41.542+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:41.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:41.300+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:41.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:41.420+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:41.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:41.491+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:41.881 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:41 smithi204 bash[72592]: time="2024-03-24T00:52:41Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2 in 30 seconds, resorting to SIGKILL" 2024-03-24T00:52:41.881 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:41 smithi204 podman[72592]: 2024-03-24 00:52:41.71998639 +0000 UTC m=+30.156512093 container died 7739ddd5d91174d101c1d3f0f7199f96d80e312c1ab3e3bb99b1fa2d7d94c949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:52:41.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:41.585+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:42.153 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 podman[72592]: 2024-03-24 00:52:42.061791581 +0000 UTC m=+30.498317283 container cleanup 7739ddd5d91174d101c1d3f0f7199f96d80e312c1ab3e3bb99b1fa2d7d94c949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:52:42.153 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 bash[72592]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2 2024-03-24T00:52:42.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:42.019+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:42.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:42.312+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:42.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:42.148+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:42.523 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:42.328+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:42.524 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 podman[72621]: 2024-03-24 00:52:42.27181379 +0000 UTC m=+0.554150062 container remove 7739ddd5d91174d101c1d3f0f7199f96d80e312c1ab3e3bb99b1fa2d7d94c949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:52:42.524 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.2.service: Main process exited, code=exited, status=137/n/a 2024-03-24T00:52:42.780 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:42.522+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:42.781 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:42.585+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:42.781 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 podman[72719]: 2024-03-24 00:52:42.612784353 +0000 UTC m=+0.018823837 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:52:42.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:42.453+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:42.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:42.400+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:43.071 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:42 smithi204 podman[72719]: 2024-03-24 00:52:42.78107102 +0000 UTC m=+0.187110482 container create 0e8df341cf45b4519cb015b6d72c618e929e31bbdd5ffda02f1e6583663a1e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:52:43.072 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:43 smithi204 podman[72719]: 2024-03-24 00:52:43.01692008 +0000 UTC m=+0.422959562 container init 0e8df341cf45b4519cb015b6d72c618e929e31bbdd5ffda02f1e6583663a1e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T00:52:43.072 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:43 smithi204 podman[72719]: 2024-03-24 00:52:43.020072986 +0000 UTC m=+0.426112462 container start 0e8df341cf45b4519cb015b6d72c618e929e31bbdd5ffda02f1e6583663a1e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T00:52:43.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:43.053+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:43.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:43.335+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:43.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:43.181+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:43.520 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:43.318+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:43.521 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:43 smithi204 podman[72719]: 2024-03-24 00:52:43.117048026 +0000 UTC m=+0.523087511 container attach 0e8df341cf45b4519cb015b6d72c618e929e31bbdd5ffda02f1e6583663a1e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:52:43.521 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 24 00:52:43 smithi204 podman[72719]: 2024-03-24 00:52:43.316891321 +0000 UTC m=+0.722930793 container died 0e8df341cf45b4519cb015b6d72c618e929e31bbdd5ffda02f1e6583663a1e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:52:43.812 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:43.519+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:43.812 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:43.617+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:43.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T00:52:43.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:43.475+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:43.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:43.434+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:43.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:52:43.877 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T00:52:43.877 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T00:52:43.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.3 2024-03-24T00:52:44.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:43 smithi123 systemd[1]: Stopping Ceph osd.3 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:52:44.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:44.015+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:44.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:44.383+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:44.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:44.146+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:44.572 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:44.473+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:44.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:44.314+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:44.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:44.429+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:44.856 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:44.440+0000 7f5b30000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:52:44.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:44.440+0000 7f5b30000700 -1 osd.3 92 *** Got signal Terminated *** 2024-03-24T00:52:44.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:44.440+0000 7f5b30000700 -1 osd.3 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:52:44.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:44.439+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:45.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:44.651+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:45.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:44.990+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:45.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:45.350+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:45.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:45.123+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:45.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:45.468+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:45.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:45.285+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:45.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:45.469+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:45.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:45.433+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:46.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:45.689+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:46.357 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:46.033+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:46.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:46.325+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:46.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:46.126+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:46.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:46.446+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:46.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:46.287+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:46.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:46.471+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:46.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:46.441+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:47.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:46.696+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:47.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:47.070+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:47.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:47.360+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:47.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:47.116+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:47.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:47.241+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:47.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:47.447+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:47.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:47.483+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:47.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:47.464+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:48.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:47.697+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:48.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:48.062+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:48.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:48.365+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:48.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:48.143+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:48.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:48.237+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:48.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:48.481+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:48.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:48.512+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:48.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:48.511+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:49.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:48.710+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:49.356 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:52:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3[52934]: 2024-03-24T00:52:49.059+0000 7f5b26800700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.690946+0000 front 2024-03-24T00:51:10.690292+0000 (oldest deadline 2024-03-24T00:52:38.598491+0000) 2024-03-24T00:52:49.388 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:49.124+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:49.495 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:49.231+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:49.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:49.387+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:49.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:49.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:49.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:49.706+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:49.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:49.471+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:49.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:49.516+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:50.357 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:50.356+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:50.357 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:50.126+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:50.521 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:50.204+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:50.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:50.519+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:50.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:50.675+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:50.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:50.522+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:50.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:50.458+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:51.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:51.163+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:51.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:51.313+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:51.501 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:51.222+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:51.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:51.500+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:51.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:51.643+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:51.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:51.433+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:51.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:51.529+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:52.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:52.285+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:52.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:52.212+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:52.498 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:52.194+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:52.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:52.496+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:52.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:52.610+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:52.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:52.535+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:52.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:52.438+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:53.501 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:53.213+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:53.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:53.238+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:53.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:53.218+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:53.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:53.500+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:53.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:53.566+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:53.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:53.434+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:53.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:53.578+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:54.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:54.253+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:54.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:54.209+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:54.521 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:54.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:54.522 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:54.251+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:54.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:54.520+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:54.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:54.465+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:54.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:54.566+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:55.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:55.202+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:55.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:55.217+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:55.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:55.285+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:55.572 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:55.471+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:55.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:55.509+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:55.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:55.455+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:55.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:55.535+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:56.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:56.201+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:56.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:56.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:56.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:56.490+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:56.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:56.477+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:56.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:56.245+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:56.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:56.424+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:56.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:56.579+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:57.508 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:57.454+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:57.508 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:57.237+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:57.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:57.247+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:57.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:57.215+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:57.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:57.507+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:57.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:57.430+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:57.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:57.547+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:58.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:58.503+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:58.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:58.473+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:58.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:58.260+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:58.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:58.246+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:58.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:58.246+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:58.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:58.457+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:58.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:58.547+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:52:59.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:52:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:52:59.473+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:52:59.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:52:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:52:59.461+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:52:59.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:52:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:52:59.233+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:52:59.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:52:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:52:59.283+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:52:59.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:52:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:52:59.213+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:52:59.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:52:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:52:59.457+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:52:59.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:52:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:52:59.525+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:00.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:00.236+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:00.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:00.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:00.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:00.237+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:00.571 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:00.466+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:00.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:00.464+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:00.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:00.478+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:00.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:00.520+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:01.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:01.271+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:01.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:01.190+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:01.506 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:01.229+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:01.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:01.504+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:01.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:01.510+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:01.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:01.492+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:01.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:01.495+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:02.510 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:02.196+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:02.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:02.282+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:02.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:02.223+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:02.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:02.509+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:02.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:02.542+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:02.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:02.504+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:02.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:02.457+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:03.474 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:03.212+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:03.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:03.256+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:03.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:03.257+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:03.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:03.473+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:03.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:03.585+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:03.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:03.510+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:03.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:03.454+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:04.447 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:04.186+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:04.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:04.223+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:04.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:04.245+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:04.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:04.445+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:04.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:04.605+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:04.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:04.557+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:04.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:04.445+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:05.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:05.252+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:05.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:05.203+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:05.485 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:05.162+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:05.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:05.484+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:05.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:05.604+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:05.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:05.541+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:05.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:05.406+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:06.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:06.297+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:06.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:06.211+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:06.466 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:06.154+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:06.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:06.574+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:06.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:06.464+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:06.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:06.524+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:06.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:06.424+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:07.427 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:07.108+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:07.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:07.322+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:07.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:07.163+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:07.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:07.595+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:07.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:07.426+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:07.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:07.543+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:07.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:07.453+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:08.400 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:08.143+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:08.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:08.288+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:08.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:08.206+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:08.757 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:08 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:08.428+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:08.757 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:08 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:08.503+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:08.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:08.399+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:08.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:08.592+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:09.446 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:09.094+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:09.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:09.323+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:09.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:09.402+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:09.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:09.255+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:09.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:09.608+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:09.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:09.446+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:09.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:09.446+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:09.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:09 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:09.532+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:09.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:09 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:09.474+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:10.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:10.065+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:10.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:10.065+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:10.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:10.209+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:10.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:10.296+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:10.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:10.419+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:10.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:10.488+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:10.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:10.488+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:10.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:10.569+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:10.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:10.569+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:10.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:10.479+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:10.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:10.479+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:10.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:10.545+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:11.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:11.070+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:11.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:11.070+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:11.576 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:11.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:11.576 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:11.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:11.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:11.449+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:11.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:11.238+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:11.713 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:11.258+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:11.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:11.535+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:11.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:11.518+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:11.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:11.518+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:12.051 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:11.576+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:12.051 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:11.576+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:12.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:12.050+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:12.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:12.050+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:12.596 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:12.521+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:12.596 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:12.521+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:12.597 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:12.560+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:12.597 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:12.560+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:12.713 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:12.280+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:12.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:12.431+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:12.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:12.229+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:12.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:12.547+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:12.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:12.547+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:12.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:12.543+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:12.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:12.543+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:13.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:13.055+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:13.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:13.055+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:13.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:13.277+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:13.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:13.382+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:13.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:13.213+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:13.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:13.213+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:13.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:13.521+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:13.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:13.521+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:13.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:13.549+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:13.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:13.549+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:13.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:13.537+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:13.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:13.537+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:13.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:13.513+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:13.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:13.513+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:14.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:14.036+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:14.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:14.036+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:14.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:14.308+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:14.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:14.391+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:14.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:14.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:14.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:14.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:14.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:14.562+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:14.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:14.562+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:14.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:14.514+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:14.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:14.514+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:14.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:14.491+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:14.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:14.491+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:14.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:14 smithi123 bash[97859]: time="2024-03-24T00:53:14Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3 in 30 seconds, resorting to SIGKILL" 2024-03-24T00:53:14.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:14 smithi123 podman[97859]: 2024-03-24 00:53:14.595308283 +0000 UTC m=+30.316208876 container died e28c2b670a959e88580c0f10f14450c2b80885e3eaa976ee5e6487b7ce2e7e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T00:53:14.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:14.567+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:14.858 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:14.567+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:15.319 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:15 smithi123 podman[97859]: 2024-03-24 00:53:15.033865078 +0000 UTC m=+30.754765669 container cleanup e28c2b670a959e88580c0f10f14450c2b80885e3eaa976ee5e6487b7ce2e7e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:53:15.319 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:15 smithi123 bash[97859]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3 2024-03-24T00:53:15.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:15.024+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:15.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:15.024+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:15.606 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:15 smithi123 podman[97876]: 2024-03-24 00:53:15.318220506 +0000 UTC m=+0.710631294 container remove e28c2b670a959e88580c0f10f14450c2b80885e3eaa976ee5e6487b7ce2e7e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T00:53:15.607 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:15 smithi123 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.3.service: Main process exited, code=exited, status=137/n/a 2024-03-24T00:53:15.607 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:15.556+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:15.607 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:15.556+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:15.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:15.471+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:15.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:15.471+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:15.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:15.293+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:15.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:15.426+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:15.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:15.214+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:15.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:15.214+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:15.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:15.566+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:15.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:15.566+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:15.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:15.496+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:15.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:15.496+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:16.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:16.042+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:16.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:16.042+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:16.465 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.026619166 +0000 UTC m=+0.037138999 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:53:16.465 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.152360943 +0000 UTC m=+0.162880806 container create f9e3f0186f7d375d76d79d483cc0694ac8382112593115f1e2dbd6920a24a55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:53:16.465 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.464333893 +0000 UTC m=+0.474853759 container init f9e3f0186f7d375d76d79d483cc0694ac8382112593115f1e2dbd6920a24a55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T00:53:16.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:16.292+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:16.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:16.393+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:16.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:16.222+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:16.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:16.222+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:16.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:16.573+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:16.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:16.573+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:16.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:16.468+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:16.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:16.468+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:16.504+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:16.504+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.475171217 +0000 UTC m=+0.485691042 container start f9e3f0186f7d375d76d79d483cc0694ac8382112593115f1e2dbd6920a24a55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.537772806 +0000 UTC m=+0.548292689 container attach f9e3f0186f7d375d76d79d483cc0694ac8382112593115f1e2dbd6920a24a55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:16.578+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:16.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:16.578+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:17.255 INFO:journalctl@ceph.osd.3.smithi123.stdout:Mar 24 00:53:16 smithi123 podman[97965]: 2024-03-24 00:53:16.907135086 +0000 UTC m=+0.917654910 container died f9e3f0186f7d375d76d79d483cc0694ac8382112593115f1e2dbd6920a24a55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:53:17.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:16.994+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:17.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:16.994+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:17.597 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:17.460+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:17.597 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:17.460+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:17.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T00:53:17.712 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:17.282+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:17.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:17.351+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:17.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:17.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:17.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:17.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:17.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:53:17.771 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T00:53:17.771 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T00:53:17.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.4 2024-03-24T00:53:17.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:17.535+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:17.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:17.535+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:17.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:17.471+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:17.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:17.471+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:17.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:17.596+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:17.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:17.596+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:18.061 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:17 smithi177 systemd[1]: Stopping Ceph osd.4 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:53:18.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:17.958+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:18.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:17.958+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:18.326 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:18.059+0000 7f8bad7b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:53:18.327 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:18.059+0000 7f8bad7b4700 -1 osd.4 92 *** Got signal Terminated *** 2024-03-24T00:53:18.327 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:18.059+0000 7f8bad7b4700 -1 osd.4 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:53:18.327 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:18.326+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:18.327 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:18.241+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:18.327 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:18.241+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:18.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:18.371+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:18.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:18.506+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:18.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:18.506+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:18.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:18.497+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:18.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:18.497+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:18.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:18.632+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:18.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:18.632+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:18.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:18.460+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:18.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:18.460+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:19.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:18.946+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:19.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:18.946+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:19.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:19.372+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:19.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:19.333+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:19.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:19.201+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:19.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:19.201+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:19.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:19.502+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:19.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:19.502+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:19.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:19.483+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:19.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:19.483+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:19.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:19.628+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:19.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:19.628+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:19.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:19.422+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:19.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:19.422+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:20.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:19.915+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:20.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:19.915+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:20.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:20.326+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:20.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:20.380+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:20.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:20.168+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:20.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:20.168+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:20.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:20.531+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:20.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:20.531+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:20.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:20.504+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:20.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:20.504+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:20.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:20.410+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:20.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:20.410+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:20.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:20.642+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:20.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:20.642+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:21.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:20.938+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:21.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:20.938+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:21.462 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:21.312+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:21.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:21.357+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:21.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:21.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:21.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:21.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:21.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:21.550+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:21.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:21.550+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:21.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:21.494+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:21.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:21.494+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:21.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:21.607+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:21.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:21.607+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:21.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:21.390+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:21.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:21.390+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:22.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:21.988+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:22.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:21.988+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:22.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:22.354+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:22.463 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4[47495]: 2024-03-24T00:53:22.342+0000 7f8ba9da8700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.505513+0000 front 2024-03-24T00:51:10.505809+0000 (oldest deadline 2024-03-24T00:52:38.107854+0000) 2024-03-24T00:53:22.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:22.144+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:22.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:22.144+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:22.639 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:22.360+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:22.640 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:22.360+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:22.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:22.507+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:22.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:22.507+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:22.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:22.492+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:22.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:22.492+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:23.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:22.638+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:23.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:22.638+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:23.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:23.010+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:23.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:23.010+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:23.381 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:23.122+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:23.381 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:23.122+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:23.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:23 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:23.322+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:23.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:23 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:23.322+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:23.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:23.381+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:23.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:23.533+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:23.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:23.533+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:23.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:23.487+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:23.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:23.487+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:24.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:23 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:23.660+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:24.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:23 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:23.660+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:24.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:24.057+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:24.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:24.057+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:24.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:24.370+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:24.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:24.143+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:24.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:24.143+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:24.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:24 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:24.296+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:24.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:24 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:24.296+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:24.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:24.550+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:24.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:24.550+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:24.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:24.510+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:24.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:24.510+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:25.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:24 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:24.647+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:25.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:24 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:24.647+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:25.395 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:25.142+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:25.395 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:25.142+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:25.461 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:25.083+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:25.461 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:25.083+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:25.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:25 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:25.317+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:25.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:25 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:25.317+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:25.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:25.394+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:25.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:25.533+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:25.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:25.533+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:25.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:25.460+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:25.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:25.460+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:26.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:25 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:25.631+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:26.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:25 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:25.631+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:26.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:26.057+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:26.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:26.057+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:26.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:26.345+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:26.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:26.107+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:26.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:26.107+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:26.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:26 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:26.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:26.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:26 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:26.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:26.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:26.500+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:26.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:26.500+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:26.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:26.494+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:26.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:26.494+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:27.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:26 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:26.667+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:27.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:26 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:26.667+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:27.390 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:27.099+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:27.391 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:27.099+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:27.476 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:27.089+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:27.476 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:27.089+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:27.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:27 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:27.325+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:27.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:27 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:27.325+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:27.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:27.390+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:27.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:27.484+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:27.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:27.484+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:27.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:27.475+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:27.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:27.475+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:27.981 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:27 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:27.662+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:27.981 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:27 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:27.662+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:28.449 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:28.098+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:28.449 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:28.098+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:28.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:28.358+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:28.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:28.120+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:28.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:28.120+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:28.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:28 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:28.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:28.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:28 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:28.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:28.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:28.474+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:28.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:28.474+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:28.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:28.449+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:28.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:28.449+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:29.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:28 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:28.677+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:29.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:28 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:28.677+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:29.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:29.068+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:29.387 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:29.068+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:29.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:29.387+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:29.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:29.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:29.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:29 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:29.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:29.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:29 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:29.297+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:29.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:29.450+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:29.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:29.450+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:29.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:29.411+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:29.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:29.411+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:30.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:29 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:29.691+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:30.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:29 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:29.691+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:30.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:30.062+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:30.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:30.062+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:30.417 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:30.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:30.417 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:30.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:30.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:30 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:30.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:30.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:30 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:30.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:30.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:30.416+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:30.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:30.463+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:30.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:30.463+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:30.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:30.461+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:30.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:30.461+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:31.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:30 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:30.732+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:31.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:30 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:30.732+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:31.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:31.040+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:31.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:31.040+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:31.422 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:31.149+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:31.422 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:31.149+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:31.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:31 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:31.294+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:31.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:31 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:31.294+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:31.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:31.421+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:31.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:31.459+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:31.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:31.459+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:31.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:31.411+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:31.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:31.411+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:32.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:31 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:31.768+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:32.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:31 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:31.768+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:32.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:32.007+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:32.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:32.007+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:32.427 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:32 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:32.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:32.427 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:32 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:32.128+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:32.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:32.329+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:32.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:32.329+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:32.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:32 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:32.427+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:32.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:32.421+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:32.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:32.421+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:32.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:32.414+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:32.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:32.414+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:33.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:32.742+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:33.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:32 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:32.742+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:33.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:32.998+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:33.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:32.998+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:33.419 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:33 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:33.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:33.419 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:33 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:33.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:33.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:33.301+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:33.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:33.301+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:33.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:33 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:33.418+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:33.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:33.414+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:33.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:33.414+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:33.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:33.394+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:33.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:33.394+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:34.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:33.705+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:34.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:33 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:33.705+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:34.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:33.949+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:34.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:33.949+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:34.394 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:34 smithi177 podman[72029]: 2024-03-24 00:53:34.120545011 +0000 UTC m=+16.165179106 container died 390e1c988b66aac3b0f70a323f2c9b336ce5fc1dfb7f10d9cab2818bd45e85de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:53:34.394 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:34 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:34.144+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:34.394 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:34 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:34.144+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:34.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:34.351+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:34.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:34.351+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:34.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:34 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:34.448+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:34.713 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:34 smithi177 podman[72029]: 2024-03-24 00:53:34.484882937 +0000 UTC m=+16.529517082 container cleanup 390e1c988b66aac3b0f70a323f2c9b336ce5fc1dfb7f10d9cab2818bd45e85de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T00:53:34.713 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:34 smithi177 bash[72029]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4 2024-03-24T00:53:34.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:34.462+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:34.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:34.462+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:34.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:34.383+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:34.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:34.383+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:35.080 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:34 smithi177 podman[72046]: 2024-03-24 00:53:34.729735873 +0000 UTC m=+0.611263516 container remove 390e1c988b66aac3b0f70a323f2c9b336ce5fc1dfb7f10d9cab2818bd45e85de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T00:53:35.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:34.754+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:35.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:34 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:34.754+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:34.926+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:34.926+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:35.333 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:35 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:35.150+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:35.333 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:35 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:35.150+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:35.333 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:34.983705034 +0000 UTC m=+0.017329004 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:53:35.333 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:35.097053096 +0000 UTC m=+0.130677061 container create 4cc97b7a3123fa373a7f8030bdf5a1f006823eb382abcb997ad8055df7795365 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T00:53:35.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:35.342+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:35.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:35.343+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:35.622 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:35 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:35.482+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:35.622 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:35.333103038 +0000 UTC m=+0.366727018 container init 4cc97b7a3123fa373a7f8030bdf5a1f006823eb382abcb997ad8055df7795365 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:53:35.622 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:35.336799551 +0000 UTC m=+0.370423526 container start 4cc97b7a3123fa373a7f8030bdf5a1f006823eb382abcb997ad8055df7795365 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:53:35.622 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:35.424225472 +0000 UTC m=+0.457849469 container attach 4cc97b7a3123fa373a7f8030bdf5a1f006823eb382abcb997ad8055df7795365 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:53:35.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:35.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:35.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:35.494+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:35.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:35.338+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:35.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:35.338+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:35.918 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 24 00:53:35 smithi177 podman[72139]: 2024-03-24 00:53:35.622936138 +0000 UTC m=+0.656560114 container died 4cc97b7a3123fa373a7f8030bdf5a1f006823eb382abcb997ad8055df7795365 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:53:36.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:35.745+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:36.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:35 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:35.745+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:36.197 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T00:53:36.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:53:36.238 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T00:53:36.238 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T00:53:36.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.5 2024-03-24T00:53:36.263 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:35.973+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:36.263 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:35.973+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:36.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:36 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:36.200+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:36.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:36 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:36.200+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:36.539 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:36.325+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:36.539 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:36.325+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:36.539 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 systemd[1]: Stopping Ceph osd.5 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:53:36.539 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:36.447+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:36.539 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:36.447+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:36.787 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:36.359+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:36.787 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:36.359+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:36.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:36.538+0000 7f491a142700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:53:36.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:36.538+0000 7f491a142700 -1 osd.5 92 *** Got signal Terminated *** 2024-03-24T00:53:36.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:36.538+0000 7f491a142700 -1 osd.5 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:53:36.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:36 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:36.465+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:37.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:36.786+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:37.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:36 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:36.786+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:37.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:36.963+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:37.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:36.963+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:37.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:37 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:37.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:37.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:37 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:37.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:37.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:37.317+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:37.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:37.317+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:37.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:37.363+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:37.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:37.363+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:37.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:37.436+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:37.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:37.436+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:37.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:37 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:37.496+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:38.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:37.774+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:38.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:37 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:37.774+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:38.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:37.934+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:38.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:37.934+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:38.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:38 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:38.156+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:38.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:38 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:38.156+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:38.471 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:38.340+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:38.471 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:38.340+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:38.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:38.386+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:38.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:38.386+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:38.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:38.458+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:38.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:38.458+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:38.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:38 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:38.485+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:39.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:38.749+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:39.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:38 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:38.749+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:39.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:38.946+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:39.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:38.947+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:39.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:39.123+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:39.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:39.123+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:39.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:39.340+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:39.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:39.340+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:39.736 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:39 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:39.496+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:39.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:39.430+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:39.822 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:39.430+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:39.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:39.420+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:39.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:39.420+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:40.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:39.711+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:40.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:39 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:39.711+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:40.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:39.955+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:40.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:39 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:39.955+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:40.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:40 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:40.074+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:40.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:40 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:40.074+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:40.680 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:40.356+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:40.680 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:40.356+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:40.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:40.386+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:40.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:40.386+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:40.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:40.433+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:40.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:40.433+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:40.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:40 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:40.504+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:41.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:40.679+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:41.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:40 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:40.679+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:41.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:40.913+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:41.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:40 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:40.913+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:41.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:41 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:41.096+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:41.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:41 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:41.096+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:41.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:41.325+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:41.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:41.325+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:41.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:41.405+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.486555+0000 front 2024-03-24T00:51:08.486502+0000 (oldest deadline 2024-03-24T00:52:36.489865+0000) 2024-03-24T00:53:41.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5[48248]: 2024-03-24T00:53:41.405+0000 7f4916736700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.590987+0000 front 2024-03-24T00:51:40.590895+0000 (oldest deadline 2024-03-24T00:53:09.194067+0000) 2024-03-24T00:53:41.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:41.397+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:41.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:41.397+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:41.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:41 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:41.523+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:42.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:41.674+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:42.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:41 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:41.674+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:42.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:41.918+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:42.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:41 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:41.918+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:42.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:42.139+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:42.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:42.139+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:42.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:42.327+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:42.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:42.327+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:42.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:42.374+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:42.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:42.374+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:42.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:42.514+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:42.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:42 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:42.514+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:43.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:42.684+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:43.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:42.684+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:43.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:42.906+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:43.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:42 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:42.906+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:43.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:43.172+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:43.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:43.172+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:43.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:43.314+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:43.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:43.314+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:43.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:43.314+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:43.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:43.327+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:43.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:43.327+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:43.822 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:43.327+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:43.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:43.513+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:43.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:43 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:43.513+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:44.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:43.711+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:44.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:43 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:43.711+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:44.282 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:43.858+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:44.282 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:43.858+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:44.282 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:43 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:43.858+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:44.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:44.157+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:44.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:44.157+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:44.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:44.157+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:44.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:44.282+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:44.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:44.282+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:44.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:44.282+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:44.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:44.354+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:44.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:44.354+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:44.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:44.354+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:44.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:44.552+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:44.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:44 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:44.552+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:45.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:44.674+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:45.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:44.674+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:45.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:44 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:44.674+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:45.299 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:44.896+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:45.300 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:44.896+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:45.300 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:44 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:44.896+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:45.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:45.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:45.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:45.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:45.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:45.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:45.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:45.299+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:45.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:45.299+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:45.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:45.299+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:45.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:45.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:45.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:45.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:45.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:45.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:45.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:45.586+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:45.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:45 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:45.586+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:46.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:45.695+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:46.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:45.695+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:46.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:45 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:45.695+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:46.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:45.919+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:46.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:45.919+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:46.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:45 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:45.919+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:46.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:46.183+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:46.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:46.183+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:46.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:46.183+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:46.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:46.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:46.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:46.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:46.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:46.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:46.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:46.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:46.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:46.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:46.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:46.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:46.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:46.593+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:46.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:46 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:46.593+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:47.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:46.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:47.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:46.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:47.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:46.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:47.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:46.968+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:47.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:46.968+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:47.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:46 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:46.968+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:47.604 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:47.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:47.605 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:47.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:47.605 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:47.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:47.607 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:47.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:47.607 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:47.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:47.607 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:47.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:47.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:47.354+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:47.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:47.354+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:47.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:47 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:47.354+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:47.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:47.603+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:47.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:47.604+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:47.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:47 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:47.604+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:47.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:47.607+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:47.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:47 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:47.607+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:48.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.010+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:48.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.010+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:48.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.010+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:48.579 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:48.272+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:48.579 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:48.272+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:48.580 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:48.272+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:48.588 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:48.215+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:48.588 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:48.215+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:48.588 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:48.215+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:48.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:48.322+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:48.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:48.322+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:48.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:48.322+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:48.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:48.579+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:48.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:48.579+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:48.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:48 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:48.579+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:48.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:48.588+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:48.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:48 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:48.588+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:49.286 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.963+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:49.286 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.963+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:49.286 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:48 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:48.963+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:49.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:49.208+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:49.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:49.208+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:49.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:49.208+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:49.546 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:49.242+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:49.546 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:49.242+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:49.546 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:49.242+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:49.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:49.285+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:49.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:49.285+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:49.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:49.285+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:49.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:49.545+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:49.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:49.545+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:49.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:49 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:49.545+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:49.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:49.600+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:49.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:49 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:49.600+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:50.197 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:49 smithi204 podman[72840]: 2024-03-24 00:53:49.933977768 +0000 UTC m=+13.514959477 container died 39040478eb1da301917bb0cc0e2c04aeaa0c89c9a20abf27e7a88f827c0eed33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T00:53:50.198 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:49.960+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:50.198 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:49.960+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:50.198 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:49 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:49.960+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:50.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:50.256+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:50.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:50.256+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:50.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:50.256+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:50.547 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:50 smithi204 podman[72840]: 2024-03-24 00:53:50.296616909 +0000 UTC m=+13.877598598 container cleanup 39040478eb1da301917bb0cc0e2c04aeaa0c89c9a20abf27e7a88f827c0eed33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T00:53:50.547 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:50 smithi204 bash[72840]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5 2024-03-24T00:53:50.547 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:50.260+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:50.547 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:50.260+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:50.547 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:50.260+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:50.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:50.526+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:50.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:50.526+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:50.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:50.526+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:50.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:50.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:50.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:50.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:50.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:50 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:50.290+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:50.821 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:50 smithi204 podman[72857]: 2024-03-24 00:53:50.547770856 +0000 UTC m=+0.618248893 container remove 39040478eb1da301917bb0cc0e2c04aeaa0c89c9a20abf27e7a88f827c0eed33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:53:50.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:50.608+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:50.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:50 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:50.608+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:50.930+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:50.930+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:50 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:50.930+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:50 smithi204 podman[72949]: 2024-03-24 00:53:50.767624753 +0000 UTC m=+0.018192233 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:50 smithi204 podman[72949]: 2024-03-24 00:53:50.882023995 +0000 UTC m=+0.132591479 container create 8ab6c5d84aea674ce6a6611fd68385710fb4ac7de561c8541d8653ca5e0d0339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:53:51.151 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:51 smithi204 podman[72949]: 2024-03-24 00:53:51.084301218 +0000 UTC m=+0.334868698 container init 8ab6c5d84aea674ce6a6611fd68385710fb4ac7de561c8541d8653ca5e0d0339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T00:53:51.152 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:51 smithi204 podman[72949]: 2024-03-24 00:53:51.087413725 +0000 UTC m=+0.337981209 container start 8ab6c5d84aea674ce6a6611fd68385710fb4ac7de561c8541d8653ca5e0d0339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:53:51.152 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:51 smithi204 podman[72949]: 2024-03-24 00:53:51.150876839 +0000 UTC m=+0.401444335 container attach 8ab6c5d84aea674ce6a6611fd68385710fb4ac7de561c8541d8653ca5e0d0339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:53:51.572 INFO:journalctl@ceph.osd.5.smithi204.stdout:Mar 24 00:53:51 smithi204 podman[72949]: 2024-03-24 00:53:51.377369438 +0000 UTC m=+0.627936910 container died 8ab6c5d84aea674ce6a6611fd68385710fb4ac7de561c8541d8653ca5e0d0339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T00:53:51.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:51.306+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:51.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:51.306+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:51.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:51.306+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:51.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:51.552+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:51.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:51.552+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:51.606 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:51.552+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:51.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:51.312+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:51.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:51.312+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:51.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:51 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:51.312+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:51.643 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:51.231+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:51.643 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:51.231+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:51.643 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:51.231+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:51.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:51.643+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:51.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:51 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:51.643+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:51.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T00:53:52.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:53:52.056 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T00:53:52.056 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T00:53:52.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.6 2024-03-24T00:53:52.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:51.953+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:52.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:51.953+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:52.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:51 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:51.953+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:52.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:52.347+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:52.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:52.347+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:52.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:52.347+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:52.583 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 systemd[1]: Stopping Ceph osd.6 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:53:52.583 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:52.360+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:52.584 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:52.361+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:52.584 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:52.361+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:52.618 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:52.255+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:52.618 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:52.255+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:52.618 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:52.255+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:52.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.583+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:52.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.583+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:52.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.583+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:52.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.615+0000 7f0d55200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:53:52.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.615+0000 7f0d55200700 -1 osd.6 92 *** Got signal Terminated *** 2024-03-24T00:53:52.858 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:52.615+0000 7f0d55200700 -1 osd.6 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:53:52.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:52.618+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:52.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:52 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:52.618+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:53.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:52.998+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:53.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:52.998+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:53.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:52 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:52.998+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:53.571 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:53.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:53.572 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:53.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:53.572 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:53.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:53.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:53.377+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:53.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:53.377+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:53.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:53 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:53.377+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:53.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:53.622+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:53.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:53.622+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:53.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:53.622+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:53.858 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:53.396+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:53.858 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:53.396+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:53.858 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:53 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:53.396+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:53.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:53.571+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:53.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:53 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:53.571+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:54.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:54.001+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:54.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:54.001+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:54.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:54.001+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:54.599 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:54.229+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:54.600 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:54.229+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:54.600 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:54.229+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:54.602 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:54.351+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:54.602 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:54.351+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:54.602 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:54.351+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:54.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:54.346+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:54.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:54.346+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:54.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:54 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:54.346+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:54.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:54.601+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:54.898 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:54.601+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:54.898 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:54 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:54.601+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:54.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:54.600+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:54.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:54 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:54.600+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:55.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:55.013+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:55.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:55.013+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:55.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:55.013+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:55.557 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:55.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:55.557 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:55.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:55.557 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:55.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:55.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:55.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:55.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:55.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:55.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:55.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:55.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:55.332+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:55.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:55.333+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:55.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:55 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:55.333+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:55.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:55.555+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:55.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:55 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:55.555+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:56.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:55.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:56.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:55.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:56.107 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:55 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:55.649+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:56.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:56.041+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:56.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:56.041+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:56.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:56.041+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:56.559 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:56.245+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:56.559 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:56.245+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:56.559 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:56.245+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:56.604 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:56.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:56.604 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:56.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:56.604 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:56.313+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:56.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:56.332+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:56.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:56.332+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:56.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:56 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:56.332+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:56.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:56.603+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:56.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:56.603+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:56.857 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:56 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:56.603+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:56.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:56.558+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:56.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:56 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:56.558+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:57.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:57.006+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:57.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:57.006+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:57.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:57.006+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:57.578 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:57.219+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:57.578 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:57.219+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:57.578 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:57.219+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:57.585 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:57.303+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:57.585 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:57.303+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:57.585 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:57.303+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:57.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:57.360+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:57.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:57.360+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:57.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:57 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:57.360+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:57.858 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:57.585+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:07.321202+0000 front 2024-03-24T00:51:07.321293+0000 (oldest deadline 2024-03-24T00:52:41.227340+0000) 2024-03-24T00:53:57.858 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:57.585+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.528961+0000 front 2024-03-24T00:51:43.528676+0000 (oldest deadline 2024-03-24T00:53:12.035250+0000) 2024-03-24T00:53:57.858 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:57 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6[59892]: 2024-03-24T00:53:57.585+0000 7f0d50a00700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.837180+0000 front 2024-03-24T00:52:14.837092+0000 (oldest deadline 2024-03-24T00:53:43.746265+0000) 2024-03-24T00:53:57.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:57.577+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:57.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:57 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:57.577+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:58.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:58.043+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:58.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:58.043+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:58.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:58.043+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:58.392 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:58 smithi123 podman[98094]: 2024-03-24 00:53:58.11083109 +0000 UTC m=+5.611808678 container died 95cf87b147067762f1e8ea89c08acd7836a95e193580c66d3c8aa6368abc9d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:53:58.392 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:58.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:58.392 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:58.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:58.392 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:58.320+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:58.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:58.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:58.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:58.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:58.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:58.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:58.808 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:58 smithi123 podman[98094]: 2024-03-24 00:53:58.516482366 +0000 UTC m=+6.017459989 container cleanup 95cf87b147067762f1e8ea89c08acd7836a95e193580c66d3c8aa6368abc9d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:53:58.808 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:58 smithi123 bash[98094]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6 2024-03-24T00:53:58.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:58.383+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:58.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:58.383+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:58.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:58 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:58.383+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:58.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:58.572+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:58.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:58 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:58.572+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:53:59.106 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:58 smithi123 podman[98113]: 2024-03-24 00:53:58.807963402 +0000 UTC m=+0.681169583 container remove 95cf87b147067762f1e8ea89c08acd7836a95e193580c66d3c8aa6368abc9d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:53:59.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:59.072+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:53:59.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:59.072+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:53:59.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:53:59.072+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:53:59.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:59.159+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:53:59.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:59.160+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:53:59.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:53:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:53:59.160+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:53:59.595 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:59.285+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:53:59.595 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:59.285+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:53:59.596 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:53:59 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:53:59.285+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:53:59.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:59.422+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:53:59.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:59.422+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:53:59.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:53:59 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:53:59.422+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:53:59.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:59 smithi123 podman[98205]: 2024-03-24 00:53:59.499581481 +0000 UTC m=+0.033562840 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:53:59.856 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:59 smithi123 podman[98205]: 2024-03-24 00:53:59.667325253 +0000 UTC m=+0.201306659 container create 407eb4878a957d1f699615370e4ed03207a2e7faa0ab3250ec1d1762bf236ad8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:53:59.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:59.606+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:53:59.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:53:59 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:53:59.606+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:00.269 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:59 smithi123 podman[98205]: 2024-03-24 00:53:59.937559164 +0000 UTC m=+0.471540591 container init 407eb4878a957d1f699615370e4ed03207a2e7faa0ab3250ec1d1762bf236ad8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T00:54:00.269 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:53:59 smithi123 podman[98205]: 2024-03-24 00:53:59.948570017 +0000 UTC m=+0.482551426 container start 407eb4878a957d1f699615370e4ed03207a2e7faa0ab3250ec1d1762bf236ad8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:54:00.269 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:54:00 smithi123 podman[98205]: 2024-03-24 00:54:00.019418344 +0000 UTC m=+0.553399727 container attach 407eb4878a957d1f699615370e4ed03207a2e7faa0ab3250ec1d1762bf236ad8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:54:00.389 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:00.108+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:00.390 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:00.108+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:00.390 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:00.108+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:00.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:00.205+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:00.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:00.205+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:00.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:00.205+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:00.607 INFO:journalctl@ceph.osd.6.smithi123.stdout:Mar 24 00:54:00 smithi123 podman[98205]: 2024-03-24 00:54:00.373009899 +0000 UTC m=+0.906991274 container died 407eb4878a957d1f699615370e4ed03207a2e7faa0ab3250ec1d1762bf236ad8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T00:54:00.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:00.268+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:00.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:00.268+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:00.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:00 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:00.268+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:00.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:00.389+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:00.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:00.389+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:00.821 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:00 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:00.389+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:00.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:00.565+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:00.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:00 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:00.565+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:01.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T00:54:01.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:01.169 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T00:54:01.169 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T00:54:01.170 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.7 2024-03-24T00:54:01.461 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 systemd[1]: Stopping Ceph osd.7 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:01.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:01.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:01.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:01.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:01.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:01.220+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:01.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:01.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:01.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:01.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:01.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:01.344+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:01.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:01.144+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:01.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:01.144+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:01.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:01 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:01.144+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:01.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:01.277+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:01.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:01.277+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:01.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:01 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:01.277+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:01.712 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:01.461+0000 7f7ff448d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:01.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:01.461+0000 7f7ff448d700 -1 osd.7 92 *** Got signal Terminated *** 2024-03-24T00:54:01.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:01.461+0000 7f7ff448d700 -1 osd.7 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:54:01.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:01.566+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:01.713 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:01 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:01.566+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:02.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:02.193+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:02.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:02.193+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:02.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:02.193+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:02.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:02.308+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:02.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:02.308+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:02.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:02.308+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:02.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:02.100+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:02.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:02.100+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:02.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:02 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:02.100+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:02.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:02.266+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:02.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:02.266+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:02.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:02 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:02.266+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:02.966 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:02.596+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:02.966 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:02 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:02.596+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:03.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:03.284+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:03.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:03.284+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:03.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:03.284+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:03.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:03.110+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:03.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:03.110+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:03.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:03 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:03.110+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:03.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:03.252+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:03.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:03.252+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:03.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:03 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:03.252+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:03.623 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:03.235+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:03.623 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:03.235+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:03.623 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:03.235+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:03.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:03.623+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:03.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:03.623+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:04.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:04.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:04.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:04.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:04.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:04.199+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:04.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:04.239+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:04.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:04.239+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:04.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:04.239+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:04.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:04.120+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:04.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:04.120+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:04.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:04 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:04.120+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:04.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:04.212+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:04.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:04.212+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:04.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:04 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:04.212+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:04.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:04.641+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:04.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:04 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:04.641+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:05.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:05.229+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:05.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:05.229+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:05.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:05.229+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:05.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:05.165+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:05.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:05.165+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:05.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:05 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:05.165+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:05.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:05.183+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:05.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:05.183+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:05.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:05 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:05.183+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:05.608 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:05.225+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:05.608 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:05.225+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:05.608 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:05.225+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:05.962 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:05.608+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:40.145189+0000 front 2024-03-24T00:51:40.145207+0000 (oldest deadline 2024-03-24T00:53:09.249048+0000) 2024-03-24T00:54:05.963 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:05 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7[54497]: 2024-03-24T00:54:05.608+0000 7f7ff0a81700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.650445+0000 front 2024-03-24T00:52:12.650415+0000 (oldest deadline 2024-03-24T00:53:41.953423+0000) 2024-03-24T00:54:06.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:06.242+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:06.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:06.242+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:06.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:06.242+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:06.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:06.153+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:06.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:06.153+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:06.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:06 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:06.153+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:06.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:06.177+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:06.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:06.177+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:06.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:06 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:06.177+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:06.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:06.274+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:06.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:06.274+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:06.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:06.274+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:07.211 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:06 smithi177 podman[72256]: 2024-03-24 00:54:06.83774912 +0000 UTC m=+5.474221276 container died 9b1e261e382e0cf2be22ef6a97e46c9d1eafdd4756880c8fca5d2aa18fdd5d68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T00:54:07.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:07 smithi177 podman[72256]: 2024-03-24 00:54:07.310517051 +0000 UTC m=+5.946989225 container cleanup 9b1e261e382e0cf2be22ef6a97e46c9d1eafdd4756880c8fca5d2aa18fdd5d68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T00:54:07.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:07 smithi177 bash[72256]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7 2024-03-24T00:54:07.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:07.261+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:07.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:07.261+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:07.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:07 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:07.261+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:07.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:07.233+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:07.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:07.233+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:07.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:07.233+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:07.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:07.184+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:07.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:07.184+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:07.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:07 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:07.184+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:07.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:07.207+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:07.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:07.207+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:07.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:07 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:07.207+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:07.874 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:07 smithi177 podman[72274]: 2024-03-24 00:54:07.569997772 +0000 UTC m=+0.731507354 container remove 9b1e261e382e0cf2be22ef6a97e46c9d1eafdd4756880c8fca5d2aa18fdd5d68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:54:08.156 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:07 smithi177 podman[72362]: 2024-03-24 00:54:07.778361697 +0000 UTC m=+0.018020310 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:54:08.156 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:07 smithi177 podman[72362]: 2024-03-24 00:54:07.87887954 +0000 UTC m=+0.118538195 container create 39b25b8218a4837ad05197e7c8fce609a8866df72f3e19ae3b5d88804dbb66e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:54:08.156 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:08 smithi177 podman[72362]: 2024-03-24 00:54:08.081188568 +0000 UTC m=+0.320847167 container init 39b25b8218a4837ad05197e7c8fce609a8866df72f3e19ae3b5d88804dbb66e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T00:54:08.156 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:08 smithi177 podman[72362]: 2024-03-24 00:54:08.084433276 +0000 UTC m=+0.324091874 container start 39b25b8218a4837ad05197e7c8fce609a8866df72f3e19ae3b5d88804dbb66e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:54:08.156 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:08 smithi177 podman[72362]: 2024-03-24 00:54:08.155950193 +0000 UTC m=+0.395608804 container attach 39b25b8218a4837ad05197e7c8fce609a8866df72f3e19ae3b5d88804dbb66e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:54:08.462 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 24 00:54:08 smithi177 podman[72362]: 2024-03-24 00:54:08.368266989 +0000 UTC m=+0.607925582 container died 39b25b8218a4837ad05197e7c8fce609a8866df72f3e19ae3b5d88804dbb66e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T00:54:08.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:08.286+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:08.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:08.286+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:08.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:08 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:08.286+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:08.508 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:08 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:08.222+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:08.509 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:08 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:08.222+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:08.509 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:08 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:08.222+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:08.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:08.186+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:08.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:08.186+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:08.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:08.186+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:08.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:08.175+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:08.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:08.175+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:08.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:08 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:08.175+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:08.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T00:54:08.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:08.989 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T00:54:08.989 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T00:54:08.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.8 2024-03-24T00:54:09.290 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 systemd[1]: Stopping Ceph osd.8 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:09.290 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.166+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:09.290 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.166+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:09.291 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.166+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:09.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:09.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:09.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:09.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:09.291 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:09.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:09.531 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:09 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:09.221+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:09.532 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:09 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:09.221+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:09.532 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:09 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:09.221+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:09.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.289+0000 7f7ab6d17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:09.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.289+0000 7f7ab6d17700 -1 osd.8 92 *** Got signal Terminated *** 2024-03-24T00:54:09.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:09 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:09.289+0000 7f7ab6d17700 -1 osd.8 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:54:09.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:09.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:09.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:09.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:09.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:09 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:09.240+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:10.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:10.236+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:10.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:10.236+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:10.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:10.236+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:10.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:10.170+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:10.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:10.170+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:10.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:10.170+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:10.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:10.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:10.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:10.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:10.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:10 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:10.142+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:10.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:10.220+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:10.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:10.220+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:10.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:10 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:10.220+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:11.356 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:11.232+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:11.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:11.232+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:11.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:11 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:11.232+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:11.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:11.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:11.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:11.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:11.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:11 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:11.189+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:11.472 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:11.215+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:11.473 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:11.215+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:11.473 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:11.215+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:11.473 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:11.109+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:11.473 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:11.109+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:11.473 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:11 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:11.109+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:12.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:12.164+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:12.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:12.164+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:12.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:12 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:12.164+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:12.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:12.173+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:12.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:12.173+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:12.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:12.173+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:12.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:12.149+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:12.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:12.149+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:12.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:12 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:12.149+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:12.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:12.189+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:12.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:12.189+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:12.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:12 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:12.189+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:13.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:13.147+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:13.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:13.147+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:13.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:13 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:13.147+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:13.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:13.210+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:13.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:13.210+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:13.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:13.210+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:13.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:13.113+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:13.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:13.113+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:13.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:13 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:13.113+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:13.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:13.149+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:13.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:13.149+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:13.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:13 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:13.149+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:14.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:14.174+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:14.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:14.174+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:14.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:14.174+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:14.571 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:14.243+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.539055+0000 front 2024-03-24T00:51:10.539222+0000 (oldest deadline 2024-03-24T00:52:36.841747+0000) 2024-03-24T00:54:14.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:14.243+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.743322+0000 front 2024-03-24T00:51:43.743386+0000 (oldest deadline 2024-03-24T00:53:09.747605+0000) 2024-03-24T00:54:14.572 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8[54990]: 2024-03-24T00:54:14.243+0000 7f7ab330b700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:12.648460+0000 front 2024-03-24T00:52:12.648418+0000 (oldest deadline 2024-03-24T00:53:42.553766+0000) 2024-03-24T00:54:14.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:14.099+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:14.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:14.099+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:14.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:14 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:14.099+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:14.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:14.112+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:14.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:14.112+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:14.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:14 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:14.112+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:15.116 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:14 smithi204 podman[73065]: 2024-03-24 00:54:14.840892731 +0000 UTC m=+5.649847285 container died 53a328917d6607c774b9ed73ebf99f158dd5a4aba3c6354e2ed3f5373661b735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T00:54:15.116 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:15.055+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:15.116 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:15.055+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:15.117 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:15 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:15.055+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:15.356 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:15.104+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:15.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:15.104+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:15.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:15 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:15.104+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:15.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:15.136+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:15.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:15.136+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:15.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:15.136+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:15.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:15 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:15.136+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:15.466 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73065]: 2024-03-24 00:54:15.206682732 +0000 UTC m=+6.015637289 container cleanup 53a328917d6607c774b9ed73ebf99f158dd5a4aba3c6354e2ed3f5373661b735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T00:54:15.466 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 bash[73065]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8 2024-03-24T00:54:15.749 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73095]: 2024-03-24 00:54:15.46610566 +0000 UTC m=+0.621213340 container remove 53a328917d6607c774b9ed73ebf99f158dd5a4aba3c6354e2ed3f5373661b735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T00:54:16.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73185]: 2024-03-24 00:54:15.75017825 +0000 UTC m=+0.102576934 container create ff010d76530e2fc3d59f2c7d00ad8df3b12889a61b9a446b1d1a45910fad8e6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:54:16.071 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73185]: 2024-03-24 00:54:15.664508207 +0000 UTC m=+0.016906925 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:54:16.072 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73185]: 2024-03-24 00:54:15.919966374 +0000 UTC m=+0.272365091 container init ff010d76530e2fc3d59f2c7d00ad8df3b12889a61b9a446b1d1a45910fad8e6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:54:16.072 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73185]: 2024-03-24 00:54:15.923173966 +0000 UTC m=+0.275572649 container start ff010d76530e2fc3d59f2c7d00ad8df3b12889a61b9a446b1d1a45910fad8e6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:54:16.072 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:15 smithi204 podman[73185]: 2024-03-24 00:54:15.99438532 +0000 UTC m=+0.346784020 container attach ff010d76530e2fc3d59f2c7d00ad8df3b12889a61b9a446b1d1a45910fad8e6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T00:54:16.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:16.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:16.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:16.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:16.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:16.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:16.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:16 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:16.179+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:16.514 INFO:journalctl@ceph.osd.8.smithi204.stdout:Mar 24 00:54:16 smithi204 podman[73185]: 2024-03-24 00:54:16.217508167 +0000 UTC m=+0.569906865 container died ff010d76530e2fc3d59f2c7d00ad8df3b12889a61b9a446b1d1a45910fad8e6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T00:54:16.514 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:16.097+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:16.514 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:16.097+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:16.514 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:16 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:16.097+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:16.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:16.122+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:16.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:16.122+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:16.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:16.122+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:16.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:16 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:16.122+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:16.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T00:54:16.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:16.830 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T00:54:16.830 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T00:54:16.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.9 2024-03-24T00:54:17.356 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:16 smithi123 systemd[1]: Stopping Ceph osd.9 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:17.356 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.111+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:17.356 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.111+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:17.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.111+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:17.357 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.111+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:17.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:17.207+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:17.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:17.207+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:17.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:17.207+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:17.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:17 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:17.207+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:17.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:17.112+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:17.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:17.112+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:17.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:17 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:17.112+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:17.856 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.471+0000 7f8ec8000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:17.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.471+0000 7f8ec8000700 -1 osd.9 92 *** Got signal Terminated *** 2024-03-24T00:54:17.857 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:17 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:17.471+0000 7f8ec8000700 -1 osd.9 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:54:18.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:18.186+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:18.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:18.186+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:18.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:18.186+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:18.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:18 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:18.186+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:18.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:18.094+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:18.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:18.094+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:18.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:18 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:18.094+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:18.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:18.142+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:18.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:18.142+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:18.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:18.142+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:18.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:18 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:18.142+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:19.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:19.202+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:19.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:19.202+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:19.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:19.202+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:19.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:19 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:19.202+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:19.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:19.098+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:19.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:19.098+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:19.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:19 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:19.098+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:19.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:19.139+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:19.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:19.139+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:19.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:19.139+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:19.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:19 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:19.139+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:20.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:20.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:20.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:20.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:20.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:20.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:20.536 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:20.187+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:20.537 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:20.187+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:20.537 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:20.187+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:20.537 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:20 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:20.187+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:20.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:20.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:20.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:20.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:20.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:20.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:20.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:20 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:20.234+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:21.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:21.033+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:21.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:21.033+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:21.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:21 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:21.033+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:21.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:21.192+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:21.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:21.192+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:21.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:21.192+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:21.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:21 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:21.192+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:21.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:21.211+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:21.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:21.211+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:21.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:21.211+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:21.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:21 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:21.211+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:22.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:22.020+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:22.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:22.020+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:22.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:22 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:22.020+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:22.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:22.177+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:22.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:22.177+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:22.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:22.177+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:22.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:22 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:22.177+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:22.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:22.254+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:10.553361+0000 front 2024-03-24T00:51:10.553166+0000 (oldest deadline 2024-03-24T00:52:38.057872+0000) 2024-03-24T00:54:22.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:22.254+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:43.662739+0000 front 2024-03-24T00:51:43.662800+0000 (oldest deadline 2024-03-24T00:53:10.172668+0000) 2024-03-24T00:54:22.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:22.254+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.975195+0000 front 2024-03-24T00:52:15.975237+0000 (oldest deadline 2024-03-24T00:53:42.683660+0000) 2024-03-24T00:54:22.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:22 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9[66483]: 2024-03-24T00:54:22.254+0000 7f8ec3800700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:45.585069+0000 front 2024-03-24T00:52:45.585183+0000 (oldest deadline 2024-03-24T00:54:15.390549+0000) 2024-03-24T00:54:23.254 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:22 smithi123 podman[98323]: 2024-03-24 00:54:22.946927427 +0000 UTC m=+5.642105047 container died be8238cb7e42a5222248ac39b0858e320be7e506fc77adb2bbb5fcaacdec9d73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:54:23.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:23.069+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:23.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:23.069+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:23.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:23 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:23.069+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:23.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:23.175+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:23.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:23.175+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:23.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:23.175+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:23.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:23 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:23.175+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:23.606 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:23 smithi123 podman[98323]: 2024-03-24 00:54:23.365150581 +0000 UTC m=+6.060328194 container cleanup be8238cb7e42a5222248ac39b0858e320be7e506fc77adb2bbb5fcaacdec9d73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:54:23.607 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:23 smithi123 bash[98323]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9 2024-03-24T00:54:24.106 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:23 smithi123 podman[98344]: 2024-03-24 00:54:23.621268915 +0000 UTC m=+0.658788944 container remove be8238cb7e42a5222248ac39b0858e320be7e506fc77adb2bbb5fcaacdec9d73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:54:24.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:24.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:24.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:24.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:24.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:24.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:24.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:24.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:24.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:24.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:24.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:24.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:24.713 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:24 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:24.221+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:24.745 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:24 smithi123 podman[98435]: 2024-03-24 00:54:24.315027876 +0000 UTC m=+0.032833405 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:54:24.746 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:24 smithi123 podman[98435]: 2024-03-24 00:54:24.433644375 +0000 UTC m=+0.151449942 container create cde5d0349516f6301215bb4adf7642f53de50ac4510a6bd018a19b8f0309628d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:54:24.746 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:24 smithi123 podman[98435]: 2024-03-24 00:54:24.745267675 +0000 UTC m=+0.463073242 container init cde5d0349516f6301215bb4adf7642f53de50ac4510a6bd018a19b8f0309628d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T00:54:25.106 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:24 smithi123 podman[98435]: 2024-03-24 00:54:24.755826932 +0000 UTC m=+0.473632462 container start cde5d0349516f6301215bb4adf7642f53de50ac4510a6bd018a19b8f0309628d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:54:25.106 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:24 smithi123 podman[98435]: 2024-03-24 00:54:24.852658965 +0000 UTC m=+0.570464552 container attach cde5d0349516f6301215bb4adf7642f53de50ac4510a6bd018a19b8f0309628d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:54:25.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:25.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:25.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:25.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:25.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:25 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:25.051+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:25.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:25.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:25.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:25.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:25.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:25.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:25.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:25 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:25.187+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:25.585 INFO:journalctl@ceph.osd.9.smithi123.stdout:Mar 24 00:54:25 smithi123 podman[98435]: 2024-03-24 00:54:25.181504103 +0000 UTC m=+0.899309670 container died cde5d0349516f6301215bb4adf7642f53de50ac4510a6bd018a19b8f0309628d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-9-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:54:25.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T00:54:26.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:26.000 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T00:54:26.000 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T00:54:26.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.10 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 systemd[1]: Stopping Ceph osd.10 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.284+0000 7fd3f2923700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.284+0000 7fd3f2923700 -1 osd.10 92 *** Got signal Terminated *** 2024-03-24T00:54:26.286 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:26 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:26.284+0000 7fd3f2923700 -1 osd.10 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:54:26.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:26.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:26.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:26.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:26.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:26 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:26.056+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:27.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:27.023+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:27.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:27.023+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:27.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:27 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:27.023+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:27.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:27.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:27.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:27.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:27.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:27.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:27.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:27 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:27.113+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:28.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:28.034+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:28.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:28.034+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:28.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:28 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:28.034+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:28.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:28.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:28.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:28.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:28.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:28.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:28.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:28 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:28.127+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:29.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.000+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:29.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.000+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:29.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.000+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:29.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:29.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:29.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:29.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:29 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:29.140+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:30.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.981+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:30.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.981+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:30.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:29.981+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:30.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:30.095+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:30.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:30.095+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:30.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:30.095+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:30.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:30 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:30.095+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:31.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:30.939+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:31.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:30.939+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:31.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:30 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:30.939+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:31.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:31.098+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:06.003409+0000 front 2024-03-24T00:51:06.003313+0000 (oldest deadline 2024-03-24T00:52:38.707039+0000) 2024-03-24T00:54:31.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:31.098+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.007736+0000 front 2024-03-24T00:51:41.007784+0000 (oldest deadline 2024-03-24T00:53:13.012027+0000) 2024-03-24T00:54:31.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:31.098+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:14.012994+0000 front 2024-03-24T00:52:14.013014+0000 (oldest deadline 2024-03-24T00:53:43.816504+0000) 2024-03-24T00:54:31.463 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:31 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10[61289]: 2024-03-24T00:54:31.098+0000 7fd3eaf0f700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.123:6814 osd.3 since back 2024-03-24T00:52:44.317281+0000 front 2024-03-24T00:52:44.317258+0000 (oldest deadline 2024-03-24T00:54:15.021548+0000) 2024-03-24T00:54:32.110 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:31 smithi177 podman[72480]: 2024-03-24 00:54:31.851250833 +0000 UTC m=+5.655216820 container died f58be617d0308edf08012f154335a83a87ba078175d12450d04e93e6320c5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:54:32.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:31.955+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:32.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:31.955+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:32.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:31 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:31.955+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:32.461 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72480]: 2024-03-24 00:54:32.209778691 +0000 UTC m=+6.013744672 container cleanup f58be617d0308edf08012f154335a83a87ba078175d12450d04e93e6320c5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:54:32.461 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 bash[72480]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10 2024-03-24T00:54:32.712 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72496]: 2024-03-24 00:54:32.461019602 +0000 UTC m=+0.603460246 container remove f58be617d0308edf08012f154335a83a87ba078175d12450d04e93e6320c5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-24T00:54:33.055 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72589]: 2024-03-24 00:54:32.676357399 +0000 UTC m=+0.017698478 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:54:33.055 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72589]: 2024-03-24 00:54:32.778462899 +0000 UTC m=+0.119803988 container create 92a7146c618bd31bbb20bae0bebae2f3a3578ba48b08c56ff36bf5a3a5a0bf68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T00:54:33.055 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72589]: 2024-03-24 00:54:32.980889887 +0000 UTC m=+0.322230972 container init 92a7146c618bd31bbb20bae0bebae2f3a3578ba48b08c56ff36bf5a3a5a0bf68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T00:54:33.055 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:32 smithi177 podman[72589]: 2024-03-24 00:54:32.98427059 +0000 UTC m=+0.325611662 container start 92a7146c618bd31bbb20bae0bebae2f3a3578ba48b08c56ff36bf5a3a5a0bf68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:54:33.055 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:33 smithi177 podman[72589]: 2024-03-24 00:54:33.055482296 +0000 UTC m=+0.396823367 container attach 92a7146c618bd31bbb20bae0bebae2f3a3578ba48b08c56ff36bf5a3a5a0bf68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T00:54:33.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:32.996+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:33.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:32.996+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:33.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:32 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:32.996+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:33.462 INFO:journalctl@ceph.osd.10.smithi177.stdout:Mar 24 00:54:33 smithi177 podman[72589]: 2024-03-24 00:54:33.272963941 +0000 UTC m=+0.614305015 container died 92a7146c618bd31bbb20bae0bebae2f3a3578ba48b08c56ff36bf5a3a5a0bf68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T00:54:33.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T00:54:33.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:33.945 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T00:54:33.945 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T00:54:33.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.11 2024-03-24T00:54:34.233 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:33.949+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:34.233 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:33.949+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:34.233 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:33.949+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:34.233 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:33 smithi204 systemd[1]: Stopping Ceph osd.11 for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:34.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.232+0000 7fd6a1357700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:34.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.232+0000 7fd6a1357700 -1 osd.11 92 *** Got signal Terminated *** 2024-03-24T00:54:34.572 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.232+0000 7fd6a1357700 -1 osd.11 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:54:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.979+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.979+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.979+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:35.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:34 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:34.979+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-24T00:53:18.764320+0000 front 2024-03-24T00:53:18.764268+0000 (oldest deadline 2024-03-24T00:54:34.465777+0000) 2024-03-24T00:54:36.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:35.997+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:36.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:35.997+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:36.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:35.997+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:36.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:35 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:35.997+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-24T00:53:18.764320+0000 front 2024-03-24T00:53:18.764268+0000 (oldest deadline 2024-03-24T00:54:34.465777+0000) 2024-03-24T00:54:37.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:36.972+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:37.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:36.972+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:37.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:36.972+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:37.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:36 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:36.972+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-24T00:53:18.764320+0000 front 2024-03-24T00:53:18.764268+0000 (oldest deadline 2024-03-24T00:54:34.465777+0000) 2024-03-24T00:54:38.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:37.964+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:38.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:37.964+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:38.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:37.964+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:38.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:37 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:37.964+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-24T00:53:18.764320+0000 front 2024-03-24T00:53:18.764268+0000 (oldest deadline 2024-03-24T00:54:34.465777+0000) 2024-03-24T00:54:39.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:38.971+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-24T00:51:08.547478+0000 front 2024-03-24T00:51:08.547260+0000 (oldest deadline 2024-03-24T00:52:37.850443+0000) 2024-03-24T00:54:39.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:38.971+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-03-24T00:51:41.951270+0000 front 2024-03-24T00:51:41.951338+0000 (oldest deadline 2024-03-24T00:53:09.453985+0000) 2024-03-24T00:54:39.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:38.971+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-03-24T00:52:15.255404+0000 front 2024-03-24T00:52:15.255341+0000 (oldest deadline 2024-03-24T00:53:42.959774+0000) 2024-03-24T00:54:39.322 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:38 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11[62060]: 2024-03-24T00:54:38.971+0000 7fd699943700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-24T00:53:18.764320+0000 front 2024-03-24T00:53:18.764268+0000 (oldest deadline 2024-03-24T00:54:34.465777+0000) 2024-03-24T00:54:40.043 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:39 smithi204 podman[73290]: 2024-03-24 00:54:39.780180597 +0000 UTC m=+5.650867266 container died 547f0021367499bd7677df6379f791bdf36bb5789e10a808c6a4207e0cf19364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T00:54:40.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:40 smithi204 podman[73290]: 2024-03-24 00:54:40.124499343 +0000 UTC m=+5.995185984 container cleanup 547f0021367499bd7677df6379f791bdf36bb5789e10a808c6a4207e0cf19364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:54:40.321 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:40 smithi204 bash[73290]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11 2024-03-24T00:54:40.776 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:40 smithi204 podman[73308]: 2024-03-24 00:54:40.442464234 +0000 UTC m=+0.665844467 container remove 547f0021367499bd7677df6379f791bdf36bb5789e10a808c6a4207e0cf19364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T00:54:41.072 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:40 smithi204 podman[73401]: 2024-03-24 00:54:40.776529649 +0000 UTC m=+0.109292279 container create cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-24T00:54:41.072 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:40 smithi204 podman[73401]: 2024-03-24 00:54:40.685008479 +0000 UTC m=+0.017771144 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:54:41.072 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:41 smithi204 podman[73401]: 2024-03-24 00:54:41.020806483 +0000 UTC m=+0.353569132 container init cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:54:41.072 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:41 smithi204 podman[73401]: 2024-03-24 00:54:41.024154886 +0000 UTC m=+0.356917518 container start cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:54:41.350 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:41 smithi204 podman[73401]: 2024-03-24 00:54:41.10816366 +0000 UTC m=+0.440926305 container attach cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T00:54:41.351 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:41 smithi204 podman[73401]: 2024-03-24 00:54:41.32049319 +0000 UTC m=+0.653255827 container died cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T00:54:41.860 INFO:journalctl@ceph.osd.11.smithi204.stdout:Mar 24 00:54:41 smithi204 podman[73428]: 2024-03-24 00:54:41.840596822 +0000 UTC m=+0.514919262 container remove cafaafea86749be6b40b97a580e0d9bf7db1121d18bb781c3d051e2c8c752791 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:54:41.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T00:54:41.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:41.906 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T00:54:41.906 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T00:54:41.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.a 2024-03-24T00:54:42.325 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:42 smithi123 systemd[1]: Stopping Ceph mds.a for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:42.606 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: 2024-03-24T00:54:42.446+0000 7fab74400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:42.606 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:42 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a[73126]: 2024-03-24T00:54:42.446+0000 7fab74400700 -1 mds.a *** got signal Terminated *** 2024-03-24T00:54:45.263 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:44 smithi123 podman[98539]: 2024-03-24 00:54:44.938025819 +0000 UTC m=+2.610093853 container stop e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:54:45.263 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:44 smithi123 podman[98539]: 2024-03-24 00:54:44.966632533 +0000 UTC m=+2.638700568 container died e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T00:54:45.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:45 smithi123 podman[98539]: 2024-03-24 00:54:45.381944166 +0000 UTC m=+3.054012245 container cleanup e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-24T00:54:45.607 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:45 smithi123 bash[98539]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a 2024-03-24T00:54:46.107 INFO:journalctl@ceph.mds.a.smithi123.stdout:Mar 24 00:54:45 smithi123 podman[98565]: 2024-03-24 00:54:45.666187609 +0000 UTC m=+0.709793506 container remove e57bf069beb14672c59c21aabb1404367d699c5a78e097ddf51019b62f12091a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True) 2024-03-24T00:54:46.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T00:54:46.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:46.347 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T00:54:46.347 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T00:54:46.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.d 2024-03-24T00:54:46.606 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:46 smithi123 systemd[1]: Stopping Ceph mds.d for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:47.356 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d[72136]: 2024-03-24T00:54:46.909+0000 7f3070400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:47.357 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:46 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d[72136]: 2024-03-24T00:54:46.909+0000 7f3070400700 -1 mds.d *** got signal Terminated *** 2024-03-24T00:54:50.815 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:50 smithi123 podman[98672]: 2024-03-24 00:54:50.523849053 +0000 UTC m=+3.765220064 container died 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T00:54:51.106 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:50 smithi123 podman[98672]: 2024-03-24 00:54:50.935060583 +0000 UTC m=+4.176431614 container cleanup 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:54:51.107 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:50 smithi123 bash[98672]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d 2024-03-24T00:54:51.606 INFO:journalctl@ceph.mds.d.smithi123.stdout:Mar 24 00:54:51 smithi123 podman[98686]: 2024-03-24 00:54:51.17780404 +0000 UTC m=+0.636642259 container remove 3bf2647077ec7ab580c86fd6bdca65d1e0f9539f7f606900ffdbddaf88715c1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T00:54:51.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T00:54:51.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:51.770 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T00:54:51.770 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T00:54:51.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.g 2024-03-24T00:54:52.356 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:51 smithi123 systemd[1]: Stopping Ceph mds.g for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:52.857 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g[72741]: 2024-03-24T00:54:52.377+0000 7f93bca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:52.857 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:52 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g[72741]: 2024-03-24T00:54:52.377+0000 7f93bca00700 -1 mds.g *** got signal Terminated *** 2024-03-24T00:54:56.717 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:56 smithi123 podman[98793]: 2024-03-24 00:54:56.395579631 +0000 UTC m=+4.143970231 container died 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:54:57.072 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:56 smithi123 podman[98793]: 2024-03-24 00:54:56.837982891 +0000 UTC m=+4.586373498 container cleanup 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T00:54:57.073 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:56 smithi123 bash[98793]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g 2024-03-24T00:54:57.356 INFO:journalctl@ceph.mds.g.smithi123.stdout:Mar 24 00:54:57 smithi123 podman[98807]: 2024-03-24 00:54:57.072152896 +0000 UTC m=+0.659424438 container remove 88eb913c02077b9c99694ce0af5c66d392f9306550a73628c9f438ab43c4ee49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T00:54:57.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T00:54:57.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:54:57.641 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T00:54:57.641 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T00:54:57.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.j 2024-03-24T00:54:58.106 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:54:57 smithi123 systemd[1]: Stopping Ceph mds.j for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:54:58.606 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:54:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j[71792]: 2024-03-24T00:54:58.162+0000 7f881ee00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:54:58.606 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:54:58 smithi123 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j[71792]: 2024-03-24T00:54:58.162+0000 7f881ee00700 -1 mds.j *** got signal Terminated *** 2024-03-24T00:55:02.453 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:55:02 smithi123 podman[98913]: 2024-03-24 00:55:02.1840516 +0000 UTC m=+4.141966887 container died 1658ba157df08b20b492d0638ebb258c2268bc736ffaf714925221389a44395e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:55:02.742 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:55:02 smithi123 podman[98913]: 2024-03-24 00:55:02.5659073 +0000 UTC m=+4.523822607 container cleanup 1658ba157df08b20b492d0638ebb258c2268bc736ffaf714925221389a44395e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:55:02.742 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:55:02 smithi123 bash[98913]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j 2024-03-24T00:55:03.106 INFO:journalctl@ceph.mds.j.smithi123.stdout:Mar 24 00:55:02 smithi123 podman[98927]: 2024-03-24 00:55:02.741626534 +0000 UTC m=+0.538395128 container remove 1658ba157df08b20b492d0638ebb258c2268bc736ffaf714925221389a44395e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-j, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T00:55:03.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T00:55:03.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:03.340 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T00:55:03.340 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T00:55:03.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.b 2024-03-24T00:55:03.624 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:03 smithi177 systemd[1]: Stopping Ceph mds.b for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:03.881 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: 2024-03-24T00:55:03.623+0000 7efdba660700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:03.881 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:03 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b[66999]: 2024-03-24T00:55:03.623+0000 7efdba660700 -1 mds.b *** got signal Terminated *** 2024-03-24T00:55:05.174 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:04 smithi177 podman[72695]: 2024-03-24 00:55:04.885850657 +0000 UTC m=+1.357952850 container died 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:55:05.462 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:05 smithi177 podman[72695]: 2024-03-24 00:55:05.257068644 +0000 UTC m=+1.729170843 container cleanup 5f038e61ee0c01c63ad9ccbda962a65995410db2b6bf8a8c818fc9c5b773f9c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T00:55:05.465 INFO:journalctl@ceph.mds.b.smithi177.stdout:Mar 24 00:55:05 smithi177 bash[72695]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-b 2024-03-24T00:55:05.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T00:55:05.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:05.717 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T00:55:05.717 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T00:55:05.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.e 2024-03-24T00:55:06.045 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:05 smithi177 systemd[1]: Stopping Ceph mds.e for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:06.462 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e[66384]: 2024-03-24T00:55:06.044+0000 7ffadf3cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:06.462 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:06 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e[66384]: 2024-03-24T00:55:06.044+0000 7ffadf3cb700 -1 mds.e *** got signal Terminated *** 2024-03-24T00:55:09.702 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:09 smithi177 podman[72827]: 2024-03-24 00:55:09.43612801 +0000 UTC m=+3.496041300 container died e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T00:55:09.965 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:09 smithi177 podman[72827]: 2024-03-24 00:55:09.793624387 +0000 UTC m=+3.853537650 container cleanup e86cefa6f007d70a9414f810c3d07544279b12a19b711891b044d2e8052b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:55:09.965 INFO:journalctl@ceph.mds.e.smithi177.stdout:Mar 24 00:55:09 smithi177 bash[72827]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-e 2024-03-24T00:55:10.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T00:55:10.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:10.193 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T00:55:10.193 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T00:55:10.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.h 2024-03-24T00:55:10.462 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:10 smithi177 systemd[1]: Stopping Ceph mds.h for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:10.962 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h[66690]: 2024-03-24T00:55:10.505+0000 7fa31aacf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:10.962 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:10 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h[66690]: 2024-03-24T00:55:10.505+0000 7fa31aacf700 -1 mds.h *** got signal Terminated *** 2024-03-24T00:55:13.962 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:13 smithi177 podman[72947]: 2024-03-24 00:55:13.707147173 +0000 UTC m=+3.298301228 container died 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:55:14.310 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 podman[72947]: 2024-03-24 00:55:14.130237662 +0000 UTC m=+3.721391707 container cleanup 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T00:55:14.311 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 bash[72947]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h 2024-03-24T00:55:14.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T00:55:14.576 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 podman[72961]: 2024-03-24 00:55:14.339720528 +0000 UTC m=+0.628184237 container remove 629148218aecbb9bc71563badebf83ba49865e2eb59b29b58a3b88a740e51ecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:55:14.576 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.h.service: Deactivated successfully. 2024-03-24T00:55:14.577 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 systemd[1]: Stopped Ceph mds.h for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:55:14.577 INFO:journalctl@ceph.mds.h.smithi177.stdout:Mar 24 00:55:14 smithi177 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.h.service: Consumed 1.007s CPU time. 2024-03-24T00:55:14.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:14.587 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T00:55:14.588 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T00:55:14.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.k 2024-03-24T00:55:14.866 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:14 smithi177 systemd[1]: Stopping Ceph mds.k for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:15.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k[66064]: 2024-03-24T00:55:14.898+0000 7f1af36f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:15.212 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:14 smithi177 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k[66064]: 2024-03-24T00:55:14.898+0000 7f1af36f1700 -1 mds.k *** got signal Terminated *** 2024-03-24T00:55:19.213 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:18 smithi177 podman[73067]: 2024-03-24 00:55:18.885953168 +0000 UTC m=+4.086119373 container died 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:55:19.517 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 podman[73067]: 2024-03-24 00:55:19.332845724 +0000 UTC m=+4.533011928 container cleanup 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:55:19.517 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 bash[73067]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k 2024-03-24T00:55:19.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T00:55:19.767 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 podman[73081]: 2024-03-24 00:55:19.517158534 +0000 UTC m=+0.624884800 container remove 625105ea0357b8daf172645fa282800e395a44183df88eb4b5dd1b2f8b806aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:55:19.767 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-24T00:55:19.767 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 systemd[1]: Stopped Ceph mds.k for 082c21f6-e975-11ee-95cd-87774f69a715. 2024-03-24T00:55:19.768 INFO:journalctl@ceph.mds.k.smithi177.stdout:Mar 24 00:55:19 smithi177 systemd[1]: ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.k.service: Consumed 1.028s CPU time. 2024-03-24T00:55:19.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:19.777 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T00:55:19.778 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T00:55:19.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.c 2024-03-24T00:55:20.071 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:19 smithi204 systemd[1]: Stopping Ceph mds.c for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:20.571 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c[67508]: 2024-03-24T00:55:20.099+0000 7fd47873c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:20.571 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:20 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c[67508]: 2024-03-24T00:55:20.099+0000 7fd47873c700 -1 mds.c *** got signal Terminated *** 2024-03-24T00:55:23.321 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:22 smithi204 podman[73518]: 2024-03-24 00:55:22.939433209 +0000 UTC m=+2.946448389 container died f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:55:23.691 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:23 smithi204 podman[73518]: 2024-03-24 00:55:23.423670502 +0000 UTC m=+3.430685675 container cleanup f9a30e59b93e327e1fd6f167279ff42abe58cc3c9b57efaa9415c16bb262b1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:55:23.691 INFO:journalctl@ceph.mds.c.smithi204.stdout:Mar 24 00:55:23 smithi204 bash[73518]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-c 2024-03-24T00:55:23.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T00:55:23.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:23.885 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T00:55:23.886 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T00:55:23.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.f 2024-03-24T00:55:24.211 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:23 smithi204 systemd[1]: Stopping Ceph mds.f for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:24.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f[67197]: 2024-03-24T00:55:24.210+0000 7f081c00b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:24.571 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:24 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f[67197]: 2024-03-24T00:55:24.210+0000 7f081c00b700 -1 mds.f *** got signal Terminated *** 2024-03-24T00:55:28.486 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:28 smithi204 podman[73650]: 2024-03-24 00:55:28.227001973 +0000 UTC m=+4.122672576 container died e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T00:55:28.821 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:28 smithi204 podman[73650]: 2024-03-24 00:55:28.577336977 +0000 UTC m=+4.473007578 container cleanup e424664ecab2f5fa35399af4f25e05b721a935ff494c9d26fb44dbf16b901bd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T00:55:28.821 INFO:journalctl@ceph.mds.f.smithi204.stdout:Mar 24 00:55:28 smithi204 bash[73650]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-f 2024-03-24T00:55:29.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T00:55:29.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:29.045 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T00:55:29.046 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T00:55:29.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.i 2024-03-24T00:55:29.571 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:29 smithi204 systemd[1]: Stopping Ceph mds.i for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:29.571 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i[67813]: 2024-03-24T00:55:29.347+0000 7ff71d8d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:29.572 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:29 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i[67813]: 2024-03-24T00:55:29.347+0000 7ff71d8d1700 -1 mds.i *** got signal Terminated *** 2024-03-24T00:55:32.822 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:32 smithi204 podman[73772]: 2024-03-24 00:55:32.498397954 +0000 UTC m=+3.240774100 container died 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:55:33.230 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:32 smithi204 podman[73772]: 2024-03-24 00:55:32.822356264 +0000 UTC m=+3.564732400 container cleanup 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:55:33.230 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:32 smithi204 bash[73772]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i 2024-03-24T00:55:33.230 INFO:journalctl@ceph.mds.i.smithi204.stdout:Mar 24 00:55:33 smithi204 podman[73786]: 2024-03-24 00:55:33.023523737 +0000 UTC m=+0.523524865 container remove 4181cb41009234f59e78d061992bae422ae74059d2b6299532376fa6e5f4e81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:55:33.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T00:55:33.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:33.273 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T00:55:33.273 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T00:55:33.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.l 2024-03-24T00:55:33.571 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:33 smithi204 systemd[1]: Stopping Ceph mds.l for 082c21f6-e975-11ee-95cd-87774f69a715... 2024-03-24T00:55:34.071 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l[66889]: 2024-03-24T00:55:33.574+0000 7fe943c26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:55:34.072 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:33 smithi204 ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l[66889]: 2024-03-24T00:55:33.574+0000 7fe943c26700 -1 mds.l *** got signal Terminated *** 2024-03-24T00:55:36.276 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:36 smithi204 podman[73892]: 2024-03-24 00:55:36.017050889 +0000 UTC m=+2.533210918 container died 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:55:36.571 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:36 smithi204 podman[73892]: 2024-03-24 00:55:36.451241818 +0000 UTC m=+2.967401850 container cleanup 0155819687c40e72ed38389265e44ba5a708d00582d4243b6933800ef408fed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T00:55:36.571 INFO:journalctl@ceph.mds.l.smithi204.stdout:Mar 24 00:55:36 smithi204 bash[73892]: ceph-082c21f6-e975-11ee-95cd-87774f69a715-mds-l 2024-03-24T00:55:36.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-082c21f6-e975-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T00:55:36.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T00:55:36.902 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T00:55:36.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T00:55:37.222 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:55:42.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T00:55:42.468 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:55:45.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T00:55:45.237 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:55:47.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:55:48.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:55:48.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:55:48.109 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T00:55:48.112 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi123/crash 2024-03-24T00:55:48.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash -- . 2024-03-24T00:55:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T00:55:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-03-24T00:55:48.218 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi177/crash 2024-03-24T00:55:48.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash -- . 2024-03-24T00:55:48.263 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T00:55:48.263 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-03-24T00:55:48.265 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi204/crash 2024-03-24T00:55:48.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash -- . 2024-03-24T00:55:48.313 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/082c21f6-e975-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T00:55:48.314 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-03-24T00:55:48.315 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T00:55:48.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T00:55:48.421 INFO:teuthology.orchestra.run.smithi123.stdout:2024-03-24T00:28:48.667276+0000 mon.a (mon.0) 193 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:55:48.422 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T00:55:48.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[SEC\]' /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T00:55:48.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]' /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T00:55:48.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[WRN\]' /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T00:55:48.692 INFO:teuthology.orchestra.run.smithi123.stdout:2024-03-24T00:28:48.667276+0000 mon.a (mon.0) 193 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:55:48.693 INFO:tasks.cephadm:Compressing logs... 2024-03-24T00:55:48.693 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:55:48.697 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:55:48.700 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:55:48.728 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T00:55:48.728 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-03-24T00:55:48.728 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T00:55:48.728 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T00:55:48.730 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T00:55:48.730 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.10.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T00:55:48.730 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T00:55:48.730 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.b.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.4.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T00:55:48.730 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T00:55:48.731 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T00:55:48.733 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.i.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T00:55:48.734 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T00:55:48.743 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.h.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.e.log: 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log: 82.0% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log: 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T00:55:48.744 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log: 87.2% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log: 82.0% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log: 95.0% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.2.log: 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.c.log: 87.2% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T00:55:48.746 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log: 93.0% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T00:55:48.747 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.c.log: 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T00:55:48.747 INFO:teuthology.orchestra.run.smithi204.stderr: 94.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T00:55:48.747 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.b.log: 92.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T00:55:48.751 INFO:teuthology.orchestra.run.smithi177.stderr: 90.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T00:55:48.753 INFO:teuthology.orchestra.run.smithi204.stderr: 90.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T00:55:48.758 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T00:55:48.758 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T00:55:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T00:55:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T00:55:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T00:55:48.762 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T00:55:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T00:55:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T00:55:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T00:55:48.768 INFO:teuthology.orchestra.run.smithi123.stderr: 93.2%/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.3.log: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T00:55:48.769 INFO:teuthology.orchestra.run.smithi123.stderr: 90.3% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T00:55:48.769 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T00:55:48.770 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T00:55:48.773 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.0.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log 2024-03-24T00:55:48.774 INFO:teuthology.orchestra.run.smithi123.stderr: 94.8% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T00:55:48.775 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log: 94.8% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T00:55:48.776 INFO:teuthology.orchestra.run.smithi123.stderr: 82.5% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T00:55:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T00:55:48.778 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T00:55:48.782 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log: 87.1% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T00:55:48.782 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T00:55:48.785 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T00:55:48.791 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.d.log: /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.a.log: 94.8% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T00:55:48.797 INFO:teuthology.orchestra.run.smithi123.stderr: 90.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T00:55:49.090 INFO:teuthology.orchestra.run.smithi123.stderr: 89.9% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T00:55:49.267 INFO:teuthology.orchestra.run.smithi204.stderr: 91.8% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T00:55:49.313 INFO:teuthology.orchestra.run.smithi177.stderr: 91.8% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T00:55:49.766 INFO:teuthology.orchestra.run.smithi123.stderr: 90.2% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T00:55:54.123 INFO:teuthology.orchestra.run.smithi123.stderr: 96.6% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T00:55:54.219 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T00:55:54.354 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T00:55:54.793 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T00:55:55.319 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T00:55:55.335 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T00:55:56.585 INFO:teuthology.orchestra.run.smithi204.stderr: 93.5% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T00:55:57.022 INFO:teuthology.orchestra.run.smithi204.stderr: 93.3% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T00:55:57.024 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-24T00:55:57.024 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m8.308s 2024-03-24T00:55:57.024 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m27.882s 2024-03-24T00:55:57.024 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m1.039s 2024-03-24T00:55:57.218 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T00:55:57.648 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T00:55:57.665 INFO:teuthology.orchestra.run.smithi123.stderr: 93.1% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T00:55:58.751 INFO:teuthology.orchestra.run.smithi123.stderr: 93.1% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T00:55:58.755 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-24T00:55:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m10.028s 2024-03-24T00:55:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m36.791s 2024-03-24T00:55:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m3.094s 2024-03-24T00:55:59.043 INFO:teuthology.orchestra.run.smithi177.stderr: 93.3% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T00:56:23.318 INFO:teuthology.orchestra.run.smithi177.stderr: 95.4% -- replaced with /var/log/ceph/082c21f6-e975-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T00:56:23.319 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-24T00:56:23.319 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m34.605s 2024-03-24T00:56:23.319 INFO:teuthology.orchestra.run.smithi177.stderr:user 1m4.177s 2024-03-24T00:56:23.319 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m2.389s 2024-03-24T00:56:23.320 INFO:tasks.cephadm:Archiving logs... 2024-03-24T00:56:23.320 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi123/log 2024-03-24T00:56:23.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:56:28.359 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi177/log 2024-03-24T00:56:28.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:56:36.503 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi204/log 2024-03-24T00:56:36.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:56:40.627 INFO:tasks.cephadm:Removing cluster... 2024-03-24T00:56:40.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force 2024-03-24T00:56:40.952 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:56:41.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force 2024-03-24T00:56:41.874 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:56:42.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 082c21f6-e975-11ee-95cd-87774f69a715 --force 2024-03-24T00:56:42.395 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 082c21f6-e975-11ee-95cd-87774f69a715 2024-03-24T00:56:42.687 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T00:56:42.688 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T00:56:42.746 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T00:56:42.762 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T00:56:42.777 INFO:tasks.cephadm:Teardown complete 2024-03-24T00:56:42.777 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T00:56:42.788 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T00:56:42.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:56:42.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:56:42.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:56:42.960 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T00:56:42.970 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T00:56:42.970 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:56:42.978 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:56:42.981 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:56:42.993 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:56:42.994 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:56:42.997 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:56:42.997 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-24T00:56:42.998 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +265us[ +303us] +/- 107ms 2024-03-24T00:56:42.998 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -1014us[ -974us] +/- 54ms 2024-03-24T00:56:42.998 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv03.front.sepia.ceph.com 3 6 377 53 +1168us[+1208us] +/- 73ms 2024-03-24T00:56:42.998 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +251us[ +272us] +/- 107ms 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -1015us[ -994us] +/- 54ms 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +1156us[+1156us] +/- 73ms 2024-03-24T00:56:42.999 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:56:43.025 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +271us[ +271us] +/- 107ms 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -1008us[-1008us] +/- 55ms 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv03.front.sepia.ceph.com 3 6 377 56 +1170us[+1059us] +/- 73ms 2024-03-24T00:56:43.034 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:56:43.035 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T00:56:43.046 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T00:56:43.046 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T00:56:43.056 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T00:56:43.283 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T00:56:43.359 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T00:56:43.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:56:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:43.535 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:56:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:43.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:56:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:43.598 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T00:56:43.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T00:56:43.620 INFO:teuthology.task.internal:Duration was 2353.369256 seconds 2024-03-24T00:56:43.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T00:56:43.631 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T00:56:43.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:56:43.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:56:43.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:56:43.684 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:56:43.684 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:56:43.788 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:56:44.102 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T00:56:44.102 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-03-24T00:56:44.103 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:56:44.177 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-03-24T00:56:44.178 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:56:44.206 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-24T00:56:44.206 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:56:44.231 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T00:56:44.231 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:56:44.239 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:56:44.251 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:56:44.333 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T00:56:44.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:56:44.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:56:44.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:56:44.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T00:56:44.854 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T00:56:44.855 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:56:44.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:56:44.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:56:45.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T00:56:45.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.050 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-03-24T00:56:45.071 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-24T00:56:45.125 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-03-24T00:56:45.182 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:45.228 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:45.242 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:56:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:56:45.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T00:56:45.268 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T00:56:45.269 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi123 2024-03-24T00:56:45.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:56:45.383 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi177 2024-03-24T00:56:45.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:56:45.421 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438/remote/smithi204 2024-03-24T00:56:45.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:56:45.459 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T00:56:45.459 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T00:56:45.467 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T00:56:45.469 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T00:56:45.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T00:56:45.531 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T00:56:45.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T00:56:45.565 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T00:56:45.565 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:56:45.573 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:56:45.575 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:56:45.587 INFO:teuthology.orchestra.run.smithi177.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:56 /home/ubuntu/cephtest 2024-03-24T00:56:45.589 INFO:teuthology.orchestra.run.smithi204.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:56 /home/ubuntu/cephtest 2024-03-24T00:56:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:56 /home/ubuntu/cephtest 2024-03-24T00:56:45.627 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T00:56:45.641 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T00:56:45.665 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T00:56:45.687 DEBUG:teuthology.nuke:shortname: smithi123 2024-03-24T00:56:45.687 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:56:45.709 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.766478', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJd8CcRmbMKD+QnCRekSe6ibcqyE+2ZUHS11/ROZXhpjiZrz37YK1GE6NDNCcvcBMGDCt5vn2+yAxavfighCAM='} 2024-03-24T00:56:45.713 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-24T00:56:45.713 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:56:45.737 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-24T00:56:45.737 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:56:45.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.768068', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO1wHQDpPl2GQcDlVWxBrF7Adx1vNkAJVFxTQGHT2vC5+wJLcYHzW7O/lyCn4IfCKMy6l6h1ROCtqPjUB5+vgfA='} 2024-03-24T00:56:45.768 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-24T00:56:45.768 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:56:45.781 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:08:35.764899', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK5nwh+c/dz03Tzy3YnN8bnfvfGFcbblgZcdk9qDpoPMYBkwVJa5gaq6d/CpJcUeNTzrzVNsMXYfBaeYRQiO1w='} 2024-03-24T00:56:45.791 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-24T00:56:45.791 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:45.809 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-24T00:56:45.810 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:56:45.818 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-24T00:56:45.830 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-24T00:56:45.830 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:45.847 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-24T00:56:45.847 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:56:45.856 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-24T00:56:45.869 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-24T00:56:45.869 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:45.891 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-24T00:56:49.819 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:49.856 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:49.892 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:49.931 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-24T00:56:49.965 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-24T00:56:50.002 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-24T00:56:53.936 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:53.965 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:54.003 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:54.045 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-24T00:56:54.075 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-24T00:56:54.115 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-24T00:56:54.175 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-24T00:56:54.216 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-24T00:56:58.050 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:56:58.159 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-24T00:56:58.260 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-24T00:56:58.364 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} duration: 2353.369255542755 failure_reason: '"2024-03-24T00:28:48.667276+0000 mon.a (mon.0) 193 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: mchangir success: false 2024-03-24T00:56:58.365 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:56:58.448 INFO:teuthology.run:FAIL